KIPP is one of the most well-known and, we believe, effective charities in the United States: it has a long track record of improving students’ performance in school and Secretary of Education Arne Duncan has cited it as a model for education reform. In our recent analysis of KIPP, we’ve been surprised to learn that despite these accolades, some KIPP schools are considering cuts to core parts of the program (such as reducing Saturday school and the number of teachers available for after-school activities) and are being forced to significantly slow down the opening of new schools. KIPP Houston requires approximately $1.4 million to prevent these cuts and could use an additional $12-13 million productively to fund opening 3 new schools.

How does such a successful and acclaimed organization struggle to raise the funds necessary to continue expanding? We think a big part of the explanation comes down to the fact - which we’ve discussed before at length - that the issue of “room for more funding” (the question of how additional funding would affect a charity’s activities - i.e., the impact of the marginal donation, as opposed to the average one) is neglected by givers.

We came across the close to $15 million funding gap at KIPP Houston by probing the “room for more funding” situation for the KIPP Foundation, the national organization that we have been directing donors to. In the course of conversations on this topic, we ended up agreeing with KIPP representatives that

• The KIPP Foundation itself doesn’t have short-term “room for more funding”: it is committed to executing against its Board approved budget only. This does require a significant amount of philanthropic funding annually, but current needs pertain to future years.
• Specific KIPP regions do have room for more funding - and the KIPP Foundation doesn’t generally give grants to these regions (doing so isn’t in its model). The KIPP Foundation directed us to KIPP Houston as a specific KIPP region with a funding gap and capacity to expand.

Bottom line:

• One of the most acclaimed and effective programs in U.S. education needs $1.4 million to avoid cutting core programs and an additional $12-13 million to continue its expansion.
• This fact highlights the benefits of deeply analyzing “room for more funding,” something we’ve seen very little discussion of elsewhere in the context of charitable giving.

Details follow. (Note that we still recommend international aid organizations above U.S. organizations; this analysis is most relevant for donors who wish to focus on the U.S.)

Donors who want to support KIPP Houston directly can do so using our donate page for KIPP Houston.

Background

Over time, we’ve put more and more time and thought into the question of how to analyze a charity’s “room for more funding”, i.e., the question of how additional funding would affect a charity’s activities.

• For the early years of our project, our analysis was loose and conceptual, as we looked at basic heuristics.
• In December 2009, working with our top-rated charity VillageReach, we arrived for the first time at what we considered a “gold standard” room for more funding analysis, showing specifically how VillageReach would change its plans in response to different levels of total unrestricted revenue.
• During 2010, we started revisiting our recommended charities looking for a similar level of understanding. One of the first charities to provide it was our top-rated U.S. charity, the Nurse-Family Partnership (NFP). NFP told us that existing commitments can sustain the organization through 2015 at which point the organization would likely need additional donations to continue operations.

  We believe that donations to NFP are good: they provide a long-term safety net to an outstanding organization, and they support the cause of more effective and accountable charity more broadly. Nevertheless, we don’t think it’s fair to consider donations to NFP, today, to be the best option if there are other organizations that would use them to fill a more pressing need.

• The Knowledge Is Power Program (KIPP) is another U.S. charity we rate highly, and in response to KIPP’s question about how it could improve its rating, we responded that we’d like the best possible understanding of its room for more funding. KIPP Foundation representatives agreed to this and were extremely helpful. They were open that while the KIPP Foundation does not have the same type of long-term funding commitments that NFP has, and does require a significant amount of philanthropic funding annually, it does not have a compelling use for additional funds in the short term. But, the KIPP Foundation went on to clarify that, some of the regional networks (i.e., the actual KIPP schools) in the KIPP family do have pressing, if not urgent, financial needs.

  The KIPP Foundation does not have a mandate to directly fund KIPP schools. (For more on what it does, see our review of KIPP.) The KIPP Foundation told us that some of the regional networks have significant room for more funding. The KIPP Foundation referred us to KIPP Houston, which it believed could significantly benefit from additional funding.

KIPP Houston’s Room for More Funding needs as of July 2011

Based on this lead, we reached out to KIPP Houston, and in speaking with John Murphy, KIPP Houston’s CFO, it became apparent that it has a significant and immediate need for more funding. Specifically, KIPP Houston, which relies on money from the state for close to 85% of its annual operating budget, is planning for expected state budget cuts of 6% for the coming school year and 8.5% the following year, which would leave it with a budget shortfall of 4.7 million for the coming year.

Two notes on the data:

• The actual deficit would be slightly lower in the upcoming year and slightly higher in the subsequent year, but the average deficit over the 2 years is $4.7 million and our understanding is that KIPP would (we believe appropriately) budget for the two years together.
• We are relying heavily on KIPP Houston’s expectations for future state cuts. A two-year state budget cut of $4 billion has been passed by the legislature and according to a recent newspaper report (archived), a 6% cut across the board is expected in year one, while the subsequent year’s cut has not yet been allocated across schools, and we do not know when that will occur.

KIPP Houston told us that it expects to be able to raise an additional $1.7 million in donations (relative to what it had previously planned, a 50% increase from the previous year) and that it hopes to save $0.4 million by finding areas of potential savings (through operational efficiencies, etc.), thereby filling around $2.1 million of the funding gap itself. It could also use $1.2 million in bond income to fill the gap, though KIPP Houston told us that this would reduce its ability to expand in the near term because it had intended to put these funds toward expansion.

The remaining gap, estimated at $1.4 million, would need to be addressed through some combination of spending cuts to some of the items listed below; the final decision has not yet been made.

We asked KIPP Houston what it would do if it were able to raise additional dollars above and beyond its new fundraising goal (and close some of the gap). Mr. Murphy indicated that since many aspects of the state budget are still up in the air, these decisions have not been finalized, but were KIPP Houston able to raise additional dollars towards its budget shortfall, it would likely reinstitute “KIPP Unique” expenses and employee compensation, benefits and professional development packages first (with much of the details left to the discretion of campus directors), items we see as core to KIPP’s operations. In addition, this tightening of their budget and the use of bond income also means slowing down expansion plans in the Houston area, resulting in fewer students being served by the KIPP program than intended. Mr. Murphy explained:

“We recently were able to purchase property for our 24-26th schools and it is literally right in the heart of our more than 7,000-student waiting list. We could start things up there right away, but right now we can’t afford the construction because our capital is minimized. Those three schools could serve approximately 1,600 kids, but right now we just have to hold off and wait.”

Conclusion

As we’ve written, even a charity with proven impact isn’t necessarily a good investment - a key question is the impact of the marginal donation (not just the average one). For years we’ve recommended the Nurse-Family Partnership and the KIPP Foundation to U.S.-focused donors based on their strong case for impact, but as our analysis of “room for more funding” has sharpened, we’ve discovered that neither has a short-term need for more donations - and this in turn led us to find a specific KIPP network (KIPP Houston) that does have short-term room for more funding. (Note that we still recommend international aid organizations above U.S. organizations; this analysis is most relevant for donors who wish to focus on the U.S.).

While donating to the Nurse-Family Partnership or the KIPP Foundation does mean supporting an outstanding organization that relies on philanthropic donations to operate - and therefore is making a positive impact - we think there is a huge opportunity for a U.S.-focused donor to support organizations like KIPP Houston, for which more funds will make the difference between expansion and cutbacks.
Now the question is, how much of that nearly $15 million funding gap can we help close?

Donors who want to support KIPP Houston directly can do so using our donate page for KIPP Houston.

The New York Times recently profiled Chess in the Schools:

The Chess-in-the-Schools program has sought to foster analytical skills on the theory that these will help students succeed academically. The group teaches 20,000 children a year and calculates that it has taught 425,000 children since 1986. Children gather to learn the game at the group’s headquarters in Manhattan.

It seems like 20 years and 425,000 children is quite a lot of investment in the “theory that [chess] will help students succeed academically.” The Times feature provides a calming justification for the investment: “Chess helps promote intellectual growth and has been shown to improve academic performance.” Let’s look at the evidence for this claim.

The study we found

An early-1990s study looks at achievement test scores of chess-playing students over two years at District 9 in the Bronx. It observes that (a) the overall average reading score improved among chessplayers by about 5 percentile points, but didn’t improve among the set of remaining District 9 students; (b) 15 of 22 second-year participants improved their reading scores by some amount, while only 491 of 1118 non-participants in the district - and 245 of 655 non-participants with high reading scores, improved.

This study is riddled with major problems:

• The numbers the researchers choose to compare seem arbitrary and possibly cherry-picked. Why do the researchers look at the “percentage who improved” among second-year chessplayers but not for both years? Why do they compare the second-year students to “high-performing nonparticipants,” but not give the same comparison when looking at all students?
• The problem of selection bias is unusually obvious here. They’re comparing kids who volunteered to play chess against those who didn’t. Think of the chess club members at your school, and ask yourself if they would have been just like all the other kids had chess club not been offered. There’s no reason to think these two groups of kids are otherwise similar or would be expected to respond similarly to school.
• This is a study of somewhere between 22 and 53 students at a single district in the early 1990s. Even if the study were highly rigorous, it would still be a long way from “proof that chess helps promote intellectual growth.”

The studies we couldn’t find

The Chess-in-the-Schools website states:

In 1991 and 1996, Stuart M. Margulies, Ph.D., a noted educational psychologist, conducted two studies examining the effects of chess on children’s reading scores. The studies demonstrated that students who participated in the chess program showed improved scores on standardized tests. The gains were even greater among children with low or average initial scores. Children who were in the non-chess playing control group showed no gains.

Another study in 1999, measured the impact of chess on the emotional intelligence of fifth graders. The results of the study were striking. The overall success rate in handling real life situations with emotional intelligence was 91.4% for the children who participated in the Chess-in-the-Schools program. In contrast, those who were not involved with the chess program had an average overall success rate of only 64.4%.

We’re guessing that the study we’re looking at is an update of the 1991 study since it references no previous studies and discusses results from 1991 and 1992. We can’t find the other studies anywhere. Chess-in-the-Schools provides neither links nor citations.

Even in the best-case scenario, it’s apparently been at least a decade since the last test of the Chess-in-the-Schools model.

“Chess helps promote intellectual growth and has been shown to improve academic performance?”

In researching charities, one of the more discouraging things we’ve learned is how little support it takes for a statement like “Chess helps promote intellectual growth and has been shown to improve academic performance” to be repeated by charities, donors, and even the media.

As far as we can tell, Chess-in-the-Schools is not a demonstrated success story. It’s just been promoted and scaled up like one.

I recently posted about a new, intriguing study on the Harlem Children’s Zone. It’s now been a little over a week since David Brooks’s op-ed brought the study some major attention, and I’ve been keeping up with the reaction of other blogs. Here’s a summary:

Methodology: unusually strong

I haven’t seen any major complaints about the study’s methodology (aside from a couple of authors who appear to have raised possible concerns without having fully read the study - concerns that I don’t believe apply to it). The Social Science Statistics Blog noted it as “a nice example of careful comparisons in a non-experimental situation providing useful knowledge.”

Many studies in this area - particularly those put out by charities - have major and glaring methodological flaws/alternative hypotheses (example). We feel that this one doesn’t, which is part of what makes it so unusual and interesting.

Significance: possibly oversold

David Brooks came under a lot of criticism for his optimistic presentation of the study, stating “We may have found a remedy for the achievement gap.” Thoughts on Education Policy gives a particularly thorough overview of reasons to be cautious, including questions about whether improved test scores really point to improved opportunities and about whether this result can be replicated (”Each school has an inordinate number of things that make it unique — the Promise Academy more so than most”).

Its “What should we learn from the Promise Academy?” series (begun today) looks interesting; it is elaborating on the latter point by highlighting all the different ways in which this school is unusual.

We feel that these concerns are valid, and expressed similar concerns ourselves (here and here). However, given the weak results from past rigorous studies of education, we still feel that the results of this study bear special attention (and possible replication attempts).

Teaching to the test?

Aaron Pallas’s post on Gotham Schools raises the most interesting and worrying concern that I’ve seen.

In the HCZ Annual Report for the 2007-08 school year submitted to the State Education Department, data are presented on not just the state ELA and math assessments, but also the Iowa Test of Basic Skills. Those eighth-graders who kicked ass on the state math test? They didn’t do so well on the low-stakes Iowa Tests. Curiously, only 2 of the 77 eighth-graders were absent on the ITBS reading test day in June, 2008, but 20 of these 77 were absent for the ITBS math test. For the 57 students who did take the ITBS math test, HCZ reported an average Normal Curve Equivalent (NCE) score of 41, which failed to meet the school’s objective of an average NCE of 50 for a cohort of students who have completed at least two consecutive years at HCZ Promise Academy. In fact, this same cohort had a slightly higher average NCE of 42 in June, 2007. [Note that the study shows a huge improvement on the high-stakes test over the same time period, 2007-2008.]

Normal Curve Equivalents (NCE’s) range from 1 to 99, and are scaled to have a mean of 50 and a standard deviation of 21.06. An NCE of 41 corresponds to roughly the 33rd percentile of the reference distribution, which for the ITBS would likely be a national sample of on-grade test-takers. Scoring at the 33rd percentile is no great success story.

One possible interpretation is that cheating occurred on the higher-stakes tests, but this seems unlikely since performance was similarly strong on lower-stakes practice tests (specifics here). Another possible interpretation is that Harlem Children’s Zone teachers focused so narrowly on the high-stakes tests that they did not teach transferable skills (as Mr. Pallas implies).

We haven’t worried much about the “teaching to the test” issue to date, if only because so few interventions have shown any impact on test scores; at the very least, raising achievement test scores doesn’t appear to be easy. But this is a major concern.

Another possible interpretation is that stronger students were absent on the day of the low-stakes test, for some irrelevant reason - or that Mr. Pallas is simply misinterpreting something (I’ve only read, not vetted, his critique).

Bottom line

We know that the Fryer/Dobbie study shows an unusually encouraging result with unusual rigor. We don’t know whether it’s found a replicable way to improve important skills for disadvantaged children.

We feel that the best response to success, in an area such as this one, is not to immediately celebrate and pour in funding; it’s to investigate further.

Related posts:

• Fryer and Dobbie on Harlem Children’s Zone: what they found
• Fryer and Dobbie on the Harlem Children’s Zone: significance
• Where I stand on education, my former favorite cause
• Followup on Fryer/Dobbie study of “Harlem miracle”

Why are people so excited about one study of one charter school showing improved performance on math tests? (Our coverage of the study here).

It’s because in academic circles, improving academic performance is seen as an extremely thorny problem with a very long list of past failures. (See pages 1-2 of the paper for an overview.) The very strong default assumption is that an education program will fail to improve performance. To the point where a one-time, one-standard-deviation bump in math scores is considered (by David Brooks) to be a “miracle.”

But you’d never know it from the world of education philanthropy. Attend any fundraiser or read any annual report and all you’ll hear is stories of success.

There’s a similar split between two worlds in international philanthropy. Academics nearly all stress the challenges, the frustrations, and the sense that progress hasn’t matched expectations. Talk to a charity and you’ll hear “success, success, success.”

Many people are incredulous that we recommend so few charities. I can only guess that that’s because they’re coming from the world of fundraisers, where every charity is assumed to be a success. In our world, “recommended” is the exception, not the default.

Education used to be my favorite cause. My enthusiasm waned as I saw both the cost-effectiveness of international aid and the apparent futility of education. (Elie’s 2007 post captures many of my thoughts.) The study that I’ve been blogging about today (here and here) provides a firmer grounding for our optimism about high-intensity charter schools, and challenges the idea that there aren’t good opportunities for donors in education.

However, I’m still not ready to prioritize education again, personally. One of the things that surprised me most in studying education was not just the difficulty in finding programs that could improve academic performance, but also the complete lack of rigorous evidence that education is key to later life outcomes. I would be fascinated to see a rigorous study of how the students who benefit from excellent charter schools perform later in life - in terms of income, job satisfaction, criminal records, etc. Without evidence, I’m not convinced that raising a child’s math score raises their life prospects, especially in a way that goes beyond “signaling” (i.e., allowing them to outcompete other people due to a superior credentials).

Would putting every child in America in a good school that makes sure they can do math lead to a much better society? I used to assume it would; I’m no longer so sure, and recent information doesn’t change that.

For now, I’m going to wait and see. I’d like to see the academic reaction the Fryer and Dobbie paper on the Harlem Chilren’s Zone. If others agree about the rigor and significance of its findings, I’d like to see who steps forward to continue replicating and examining this approach. The Social Innovation Fund would seem to be one strong candidate.

In the meantime, I’m going to be putting my own money into programs that are proven and replicable and don’t have enough funding - things like tuberculosis control and distribution of insecticide-treated nets.

Perhaps, at some point, I will feel that there is an education program that meets all three of these criteria as well. At that point I may start giving to it, even if it’s many times as expensive per person as developing-world aid.

Related posts:

• Fryer and Dobbie on Harlem Children’s Zone: what they found
• Fryer and Dobbie on the Harlem Children’s Zone: significance
• Followup on Fryer/Dobbie study of “Harlem miracle”

My last post summarized a very recent paper by Fryer and Dobbie, finding large gains for charter school students in the Harlem Children’s Zone. (You can get the study here (PDF)).

I believe that this paper is an unusually important one, for reasons that the paper itself lays out very well in the first couple of pages: a wealth of existing literature has found tiny, or zero, effects from attempts to improve educational outcomes. This is truly the first time I (and, apparently, the authors) have seen a rigorously demonstrated, very large effect on math performance for any education program at all.

This study does not show that improving educational outcomes is easy. It shows that it has been done in one case by one program.

The program in question is a charter school that is extremely selective and demanding in its teacher hiring (see pages 6-7 of the study), and involves highly extended school hours (see page 6). Other high-intensity charter schools with similar characteristics have long been suspected of achieving extraordinary results (see, for example, our analysis of the Knowledge is Power Program). This study is consistent with - but more rigorous than - existing evidence suggesting that such high-intensity charter schools can accomplish what no other educational program can.

Those who doubt the value of rigorous outcomes-based analysis should consider what it has yielded in this case. Instead of scattering support among a sea of plausible-sounding programs (each backed with vivid stories and pictures of selected children), donors - and governments - are now in a position where they can focus in on one approach far more promising than the others. They can work on scaling it up across the nation and investigate whether these encouraging effects persist, and just how far the achievement gap can be narrowed. As a donor, the choice is yours: support well-meaning approaches with dubious track records (tutoring, scholarships, summer school, extracurriculars, and more), or an approach that could be the key to huge gains in academic performance.

Reasons to be cautious

The result is exciting, but:

• This is only one study, and the sample size isn’t huge (especially for the most rigorous randomization-based analysis). It’s a very new study - not even peer-reviewed yet - and it hasn’t had much opportunity to be critiqued. We look forward to seeing both critiques of it and (if the consensus is that it’s worth replicating) the results of replications.
• Observed effects are primarily on math scores. Effects on reading are encouraging but smaller. Would this improved performance translate to improvements on all aspects of the achievement gap (either causally or because the higher test scores are symptomatic of a general improvement in students’ personal development)? We don’t know.
• Just because one program worked in one place at one time doesn’t mean funding can automatically translate to more of it. Success could be a function of hard-to-replace individuals, for example. Indeed, if the consensus is that this program “works,” figuring out what about it works and how it can be extended will be an enormous challenge in itself.
• With both David Brooks and President Obama lending their enthusiasm fairly recently, the worthy mission of replicating and evaluating this program could have all the funding it needs for the near future. Individual donors should bear this in mind.

Related posts:

• Fryer and Dobbie on Harlem Children’s Zone: what they found
• Where I stand on education, my former favorite cause
• Followup on Fryer/Dobbie study of “Harlem miracle”