Over the past few weeks, we’ve seen something that we see too rarely: a debate over philanthropy.

Bill Schambra, at the invitation of the Hewlett Foundation (a funder of GiveWell), attacked the idea of “strategic philanthropy” presented in Money Well Spent (a book co-authored by Paul Brest, former Hewlett President). He characterized Money Well Spent as promoting a world in which nonprofits “sit around drinking herbal tea … filling whiteboards with guesses about why people have needs, and how to avert them altogether by getting at their root causes … [putting] more trust in the college degrees earned by the public health director than in our hard-won, practical, face-to-face understanding of our own community.” Critiques of Mr. Schambra’s argument have been posted by Paul Brest, Larry Kramer (the Hewlett Foundation’s current President) and Phil Buchanan (President of the Center for Effective Philanthropy).

Because we so rarely witness philanthropists publicly arguing with each other about how to do good philanthropy, we’re glad to see that Hewlett encouraged Mr. Schambra to put forth a critique and engaged it directly. However, we think there is a major and extremely telling missing element from this debate. The case study being debated is entirely hypothetical, involving an investigation by the “hypothetical Metro Community Foundation” of the reasons for a winter-time surge in demand for food and other necessities.

In our view, this is a sobering (and accurate) reflection on the state of foundation transparency, as well as the state of historical study of philanthropy.

Every day, people have public debates about how we should be governed, what companies we should invest in, and a wide variety of other topics, with a broad set of news and facts to draw on in challenging each other. But when it comes to how we should give, not only are there few debates, but the debates that do occur have very little to work with in terms of substantive information provided by foundations and/or independent investigators. It isn’t that there is no alternative to using hypotheticals (Money Well Spent discusses actual cases as well, and hypotheticals can have advantages in conceptual discussions); but the relative paucity of concrete examples, informative case studies, etc. is striking.

We believe that a change along this dimension would be greatly beneficial for people’s abilities to make good giving decisions, which is why we are investing in both transparency and historical study of philanthropy.

 

 

Over the years, we’ve had many exchanges along these lines:

Q: Why can’t you [research more charities / research more causes / put more effort into marketing and outreach?]
A: We don’t have enough people; we’re already stretched thin with our current priorities.
Q: What if you had enough money to hire more people?
A: We have enough money to hire more people.
Q: Then why aren’t you hiring them?
A: We’re trying.
Q: But if you have the money, what’s the downside of hiring more people? Best case, you get more done. Worst case, you get the same amount done.

It seems many people imagine that hiring, and thus capacity building, is mostly a matter of trading money for staff-hours: if we don’t have enough people to move through our to-do list, but we do have enough money to hire more, we can quickly and smoothly expand the number of staff-hours available via hiring. We think this picture of the world has important shortcomings, especially when it comes to projects like GiveWell. In our view, hiring people – whether as employees or as third-party contractors – generally means sacrificing today’s staff capacity for future staff capacity, and is always a risky (though essential) long-term investment with a difficult-to-forecast payoff. (That doesn’t mean it isn’t worthwhile – quite the opposite, as we view GiveWell’s staff as absolutely central to what it has accomplished and will accomplish.) The less well-defined and easy-to-evaluate the role one is hiring for, the more this is true, and thus the less true it is that money is the only bottleneck to expansion.

This post lays out the thinking behind our view. It focuses on Elie’s and my experiences with recruiting and managing staff (though some of the staff we’ve hired have also participated, more recently, in recruiting and managing as well). We provide relatively little in the way of empirical support for the claims in this post because we can’t go into the details of individual staff performance, and when we’ve had conversations with outsiders about these issues we generally haven’t done conversation notes for them. For readers who wish to vet our claims, one possibility is to speak to people in your own network who have experience with startup organizations (as opposed to organizations with long-established and well-defined processes and outputs), and asking whether the claims in this post ring true to them.

In a nutshell:

• In order to reach the point where employees are producing work that we can rely on and integrate into our output, senior staff need to invest significant time in training, evaluation and management.
• Investing time in making better decisions in early stages of the selection process can save time in later stages. This can include investing time directly in evaluation as well as investing time in better understanding the nature of the work we’re assigning (so as to better understand how to evaluate it).
• Hiring requires long-term predictions. There is often a substantial lag between hiring someone and having them start, and when an employee isn’t a good fit this can take time to determine. At the same time, the nature of GiveWell’s work is constantly changing. Thus, doing good evaluations of employees is both difficult and very important; the cost of an overly optimistic hire or evaluation can be significant.
• These points are less true for work that is well-defined, straightforward to evaluate, and/or similar to other work people might have on their resumes. But most of the work we need done doesn’t fit this description.
• Having adequate funds available is crucial, though not sufficient, for hiring. Cash on hand is less important than visibility into our long-term financial situation.

Growing our capacity has been a core priority of GiveWell for years. We believe it is extremely challenging and time-consuming for senior staff and often lowers rather than raises capacity in the short term, and we haven’t found a way around this fact. That said, it is also one of the most important and worthwhile things we do, and when done well can pay off massively over the long run. We feel that GiveWell’s success is ultimately mostly a function of how good a team we can build, and that GiveWell would be in a far worse position today if not for the outstanding people that have become integrated into the organization.

Employees need to be trained, evaluated and managedIn order for a new hire to save us time, it has to be the case that we rely on the new hire’s work in some sense. Research work that wasn’t careful enough, wasn’t thorough enough, or didn’t ask the questions we would ask couldn’t be used as a substantial input into our decisionmaking. Marketing work that represented GiveWell poorly could do more harm than good.

In the past, we’ve had significant problems come up when we gave an employee work and proceeded to rely on that work in some way, without first taking the time to vet the work, offer feedback, and form a thorough picture of the work’s alignment with our standards. We’ve learned to invest significantly – early in an employee’s tenure – in both training and evaluation, so that we have a clear picture of what to expect from any assignment we give, and so that we know what we can and can’t rely on the employee for given their level of ability and experience. We’ve also learned to do periodic in-depth evaluations of employees, as well as regular check-ins to discuss all aspects of their performance and assess how their responsibilities might change to find a better fit. There are some tasks that employees can do with fairly straightforward training, but there are other tasks that require deep and intuitive knowledge of GiveWell; so employees’ abilities often grow gradually and somewhat unpredictably, and constantly reassessing them pays off.

With these measures, we feel that we’re able to have a fairly good picture, at any given time, of each staff member’s strengths, weaknesses, and experience level, and this makes it possible to assign work that we’re able to rely on. Even so, as our needs and our employees’ abilities change, it can be a serious challenge to keep our picture accurate, and when our picture degrades, we often pay for it in the form of senior staff-hours (assessing the work and determining what went wrong and what adjustments can be made).

Predicting employee fit is difficultWe’ve found it to be very difficult to come to an accurate assessment of an employee’s strengths and weaknesses, particularly while recruiting/interviewing.

Traditional evaluation techniques – reading resumes, 1-2 hour interviews – have had extremely poor predictive value for us, as some of our outstanding hires have been “borderline” by such metrics while some “stars” by such metrics have turned out to be a poor fit. This has made it difficult to follow the traditional procedure of making a job posting and screening many resumes; instead, we have been trying to:

• Improve the quality of the people we consider in the first place. We have dialed up our emphasis on referrals from trusted sources as opposed to job postings; we have also constructed our jobs page to discourage casual applications and select for people who are unusually passionate about the work we do. (We believe, though with only moderate confidence, that genuine passion for GiveWell’s work is correlated with strong job performance, as much of what we look for is the ability to ask the sorts of questions we would ask in an investigation.)
• Assess people via trial hire periods whenever possible.

Exploring our network and assessing trial hires can cost significant staff time.

Even assessing an employee on the job can be very difficult. Because the nature of our work is constantly changing, we rarely have assignments that both (a) are relevant to our ongoing work; (b) can be performed without a great deal of experience on the job; (c) come with clear expectations and benchmarks in terms of what we can expect a strong new employee to produce. Without such assignments, we can’t form accurate assessments of employees and thus can’t make the most of the capacity we have. So it is sometimes the case that “lack of appropriate work to assign” is a bottleneck to hiring.

Sometimes, we believe the most productive way to make progress on hiring is to thoroughly – and personally – execute, explore and discuss the sort of work we hope to eventually outsource to employees. (This can involve multiple iterations as we find that a particular type of work turns out to be less relevant to our goals than we had anticipated.) When we thoroughly understand a type of work, we can be confident of its relevance to GiveWell, we can effectively train people on it, and we can accurately assess the quality of an employee’s work on it. We rarely try to assign work that we haven’t already done a fair amount of ourselves, and found to be relevant. Thus, hiring people to do work generally comes after – rather than instead of – doing it ourselves.

Hiring involves long-term decisions, and poor predictions can be costlyWhen we find someone we want to hire, it’s often the case that they aren’t available for a substantial amount of time. (Especially if they’re still in college, as is sometimes the case.)

In addition, we feel that hiring someone – especially when they forego other opportunities, quit their job, and/or move in order to join us – involves an implicit commitment to them, and if the relationship turns out to be a poor fit, we generally feel responsible for (a) clearly communicating that, and why, the relationship isn’t a fit; (b) doing what we can to explore whether there might be a way to improve the fit; (c) giving the employee sufficient advance notice to find another job.

Thus, the decision to hire someone often constitutes a risky “bet” that by the time they join GiveWell, their talents will still be a match for the work we need done; when we’re wrong, the process of determining this and ending the relationship can be difficult and costly. Making this sort of prediction is particularly difficult because of how quickly the nature of GiveWell’s work evolves. Starting research analysts do very different tasks today from what they did last year or the year before that.

In addition, we find all aspects of training, evaluation and management to be easier when we’re working in one office. This means that every employee affects the culture, environment, and morale of other employees. An employee who is dissatisfied or otherwise a poor fit can be a major problem.

Two pictures of hiringThe introduction of this post implies one view of hiring:

1. Lay out an assignment that we need done.
2. Post a job opportunity, collect resumes, and use a low-intensity process to evaluate candidates.
3. Hire the best candidate to do the assignment.
4. If it doesn’t go well, little is lost.

By contrast, the picture we have of hiring looks more like:

1. Execute the work we need done ourselves, and form as deep an understanding of the work as we can, so that we can later train people in parts of it and evaluate their output. When exploring a new area of research, it can often take many false starts before we can generate assignments that are stably enough defined to assign to new employees.
2. Create assignments that are both relevant to our work and relatively straightforward to evaluate.
3. Use our network to find a promising candidate and do our best to assess their long-term fit with GiveWell based on the very limited information that we have.
4. Work with the person on a trial basis, primarily aiming to train them and further assess their long-term fit with GiveWell. After each assignment, discuss what we’ve learned about the employee and how we can learn more, and communicate with the employee about future expectations.
5. If all goes well, extend a full-time offer and continue to invest heavily in training, evaluation, and management.
6. If there are problems, invest significant time in discussing and diagnosing the problems and determining whether the problems can be solved via communication with the employee or alteration of the employee’s role. If the nature of our work changes substantially (as happens regularly), make sure to revisit the employee’s fit with the work and ask what adjustments can be made.
7. If the employee does well on early, easier-to-assess work, gradually expand the scope and challenge of the employee’s responsibilities while continuing to assess the employee’s work and train the employee.
8. Ideally, the employee’s responsibilities and abilities gradually expand to the point where they become a major asset to the organization, capable of being relied on for crucial work, and in some cases capable of executing some or all of the above steps themselves (and thus increasing our “capacity for capacity building”).

The latter picture is one of risky, intensive long-time-horizon investments. There are cases in which hires or trial hires we’ve made have cost us more in training/evaluation/management time than they’ve gained us in work done, and we believe that there would be many more cases if we didn’t invest as much in early training and evaluation. However, the hires that have worked out well have eventually repaid the investment many times over, and GiveWell could not be what it is today without the strong staff we’ve built.

One of the most positive developments that can happen for GiveWell’s mission is that a hire develops to the point where we can trust them with highly challenging and important work (and that the employee stays with GiveWell for a substantial amount of time). Getting to that point requires major investment on our part in training, evaluation and management, so we pick the candidates for this carefully and invest in them heavily.

We haven’t found any shortcuts through this process (including, as discussed below, “outsourcing recruiting/management”). Building our capacity has been an extremely challenging and long-term project that has taken a great deal of co-Executive Director time and involved a great deal of learning.

When does the above dynamic hold and when doesn’t it?There are certainly cases in which work can be outsourced without all of the challenges and management investment described above. For example, we’ve outsourced our bookkeeping and our back-end web development to contractors without nearly the challenges that we’ve encountered in hiring Research Analysts (though there have still definitely been challenges).

Generally, we’d say that it’s easier to “trade money for capacity” when:

• The work we need done – and the expectations around what constitutes good work – is clearly and explicitly defined.
• The work we need done is similar enough to work that is done elsewhere that we can, relatively easily, look at someone’s resume and credentials and assess their likelihood of being able to do it.

Bookkeeping services and back-end web development (of the relatively simple variety relevant to our website) are good examples of cases where we know what we need done and how to assess the output, and we know how to find people with relevant experience.

However, most of the work we need done is far to the other end of the spectrum: it is not well-defined, it’s hard for us to say what we’re looking for in terms of output, and it’s hard to predict from someone’s resume whether they will be a good fit for it. This work includes:

• Research work including updates on top charities, investigations of potential top charities, intervention reports on things like cash transfers, conversations with experts, and shallow investigations for GiveWell Labs. While we’ve developed criteria and guidelines for this sort of work, it’s extremely far from being reduced to a formula, and anyone working on it will have to make substantial judgment calls in terms of which avenues of inquiry to pursue, which pieces of information to consider important, etc. We need to be sufficiently aligned with their judgment calls so that their work produces output that fits in with the rest of our work and our brand. We’ve experimented with recruiting people of multiple different backgrounds, and at this point we remain unaware of any particular resume item that would convince us that a person is a good fit for this role.
• Communications/outreach/marketing. Communicating effectively about GiveWell requires a deep understanding of GiveWell as well as strong communications abilities. We have worked with communications/marketing professionals in the past and have always found that getting good output requires a significant amount of our own staff time. (They can be extremely helpful, but they can’t simply take on all of the work.)
• Recruiting and management, including management of contractors (managing our bookkeeper, our web developer, etc.) We’ve toyed with the idea of trying to outsource management and recruiting, but the people we’ve spoken to about it – and our own intuitions – suggest that this would not go well. Management requires knowing our work and our needs well enough to be able to evaluate the quality of employees’ or contractors’ output; the bulk of recruiting work is evaluation (as opposed to sourcing) of candidates as well. Some of our Research Analysts have taken on management and recruiting work (including management of contractors), and in general we intend to develop Research Analysts into managers and recruiters.

The role of moneyExpanding staff obviously requires money. In general, we believe that the people who are most likely to be a good fit are people who are genuinely passionate about our mission, and for this reason, we have consistently been able to hire these people for far less than what they could earn in e.g. the for-profit sector. However, we still need to be able to pay enough so that our employees are relatively comfortable and choose to stay on rather than looking for other work.

Hiring requires long-term visibility into our future financial situation, not just (and not even primarily) cash on hand. The effort we put into recruiting at a given time is a function of how much funding we think we’ll have available for the next several years, since the full process from “starting to look for potential hires” to “having a new hire start” can be 6-12 months or more, and since full-time hires are made with the hope that someone will be with us for several years at least. Observing this has helped us to recognize the value of consistent and predictable funding, and the complexity of assessing room for more funding for certain types of organizations.

We do our best to predict our ongoing revenue on the basis of what we know about our existing donors. A donation that we are confident will be repeating for the next several years is far more valuable to us than a donation whose likelihood of repeating we can’t predict.

While money is necessary for hiring, it is not sufficient. Simply being able to fundraise does not guarantee that we will be able to expand.

ConclusionIt seems to us that much of GiveWell’s future hinges on the effectiveness of senior staff in recruiting, selecting, training, evaluating and managing the best employees possible. If we could pick one set of skills to get better at, we would pick these. Investing in this goal – both directly (recruiting, interviewing, training, evaluating, managing) and indirectly (trying to understand our own work well enough to be able to outsource it) – takes up a very large proportion of our time, rightly so given the paramount importance of the goal.

We’ve tried many things, and talked to many people, over the years. We believe the best approach involves intensive up-front investment by senior staff, rather than low-touch management. Intensive up-front investment allows us to quickly form pictures of strengths and weaknesses and invest accordingly in training and development.

We don’t believe that money, alone, can solve this problem for us, though it is an essential piece of the picture.

We’re excited about the project of making giving more analytical, more intellectual, and overall more rational. At the same time, we have mixed feelings about the project of quantifying good accomplished: of converting the impacts of all gifts into “cost per life saved” or “cost per DALY” type figures that can then be directly compared to each other.

We believe that these two projects are too often confused. Many expect (or assume) GiveWell to make all its recommendations on the basis of “cost per unit of good” formulas, and it often seems that the rare people who want to be intellectual and rational about their giving are overly interested in explicit quantification.

It seems to us that attitudes toward giving are most often classified as “emotional/intuitive” or “rational/quantitative.” We think this framing is problematic on multiple fronts. We don’t think that “rational” should be opposed to “emotional,” and we also don’t think that the very real problems of “quantitative” should be used to tar all those who seek a “rational” approach.

We propose what we think is a better framing: distinguishing “passive,” “rational” and “quantified” decisionmaking. For each, we discuss both how this style of decisionmaking approaches a “normal” purchasing decision – buying a printer – and how it approaches giving.

Passive

Passive decisionmaking follows the path of least resistance, favoring options that are salient and/or easy to choose.

A passive approach to buying a printer might involve buying the first printer that comes up in a search, or the first affordable printer that one sees in a store, or the printer suggested by a friend.

A passive approach to giving might involve supporting a charity that calls on the phone, or giving to a charity that a friend is fundraising for. It might also involve reacting directly to short-term emotional incentives: for example, a charity that tells a compelling story can induce immediate guilt/cognitive dissonance prior to a donation, and/or immediate emotional reward after a donation. Supporting such a charity can often be “easier” (in a short-term sense) than not supporting it.

We believe this describes the way most people choose charities to support.

Rational

Rational decisionmaking involves an effort to make the best possible choice (according to the decisionmaker’s values) between all viable alternatives, given constraints on how much information is available and what resources are available to gather more information.

A rational approach to buying a printer might involve a process such as

• Searching for “printer” on Amazon and using Amazon ranking, user reviews, and price as initial filters (e.g., making a list of printers that are within the price range and have strong user reviews).
• Creating a document in which the different options can be viewed side by side in terms of a variety of metrics, such as printing speed, printing quality, desk footprint, etc.
• Googling “printer,” searching for rankings and editorials, and entering noteworthy praise or criticism into the document.
• Asking friends for comments on their experiences and entering those into the document as well, as warranted.
• Looking at all the collected information, narrowing the field further based on particularly important criteria, perhaps creating some limited aggregated indices for comparison, and making a decision that considers – but does not commit to a formula regarding – the available information.
• Talking to people about one’s decision and reasoning and seeing whether one has missed any important considerations.

Of course, the amount of effort that is put in depends on how important the decision is (as well as on how much time the decisionmaker has available). In the case of a printer, the above process might or might not be overkill.

In our view, a process broadly similar to this is appropriate for choosing a charity to give to (or a cause to invest in), especially when giving a relatively large amount. We seek to:

• Consider a wide range of possible options.
• Narrow the field in successive stages, using the best heuristics we can come up with.
• Use both internal and outward-facing discussion to identify key questions that might be both (a) important to our bottom line and (b) tractable to further investigation.
• Investigate such questions and write up what we find.
• Collect all of the information into one place and thoroughly discuss it (again, both internally and externally) before making a final decision.

This approach has clear differences with the “passive” approach, but the differences are not – in my view – about “heart vs. head.” In fact, if anything, I think the rational approach is more likely to be associated with strong emotion. People often try hardest to take a “rational” approach when making their most high-stakes, emotionally important decisions; the “passive” approach is more common for decisions that are considered inconsequential.

Quantified

Quantified decisionmaking (as defined for the purposes of this post) involves committing to a universal metric for comparing all options and then formally quantifying options in terms of this metric.

A quantified approach to buying a printer might involve

• Developing a metric such as “Net present-value dollars gained” for judging printers.
• For each printer, estimating things such as
  • The expected time and convenience gained by being able to print things from home, and the present value of this time and convenience in dollars – including scenarios in which the alternative would be to print from a copy shop, scenarios in which the alternative would be to ask friends to print, and scenarios in which the alternative would be to refrain from printing a document and instead read it on the computer or rely on one’s memory.
  • The expected time lost to repairing or replacing the printer (a function of its reliability as estimated from reviews and friends’ comments), again converted to present-value dollars.
  • The monetary value of the desk space taken up by the printer.
• Creating an estimate of each relevant parameter, sometimes informed by facts and sometimes mostly based on guesswork.
• Calculating the “net present-value dollars gained” for each contending printer, making adjustments that come up on sanity checks, and purchasing the printer with the highest final score.

In our view, this approach is analogous to giving based purely on expected DALYs averted per dollar spent.

To be clear, we think this approach has some definite merits when applied to giving. We have found cost-effectiveness analysis to be highly valuable for the degree to which it brings implicit assumptions to the foreground. It often spurs debates that wouldn’t have happened otherwise, and it may provide clarity into our thinking that couldn’t be obtained in any other way. Perhaps it would accomplish similar things when applied to purchasing a printer. And this method is almost certain to be the best method when all of the relevant parameters can be estimated with high precision (this is not the case either with buying a printer or with giving to a charity, but may be the case for simpler and/or more technical questions.)

The weakness of this approach, in my view, is that it takes an enormous amount of effort to do well, and even when done well generally involves so much guesswork and uncertainty that it’s questionable whether the results should influence one’s prior beliefs. Valid, high-certainty information that should shift one’s view (for example, “this printer takes up a lot of space”) can be lost in the noise of all the guesswork used to convert the information into a unified framework (for example, converting the space taken up into dollars gained).

When using a single unified equation, one mistake – or omitted parameter – can result in a completely wrong conclusion, even if much of the other analysis that was done is sound. The “rational” approach uses implicit model combination and adjustment, and is more likely to give a good answer even when not all of the inputs are reliable. It can also be more efficient in the sense of view-shifting information gained per person-hour spent.

Conclusion

GiveWell exists to promote rational giving as opposed to passive giving. It doesn’t necessarily seek to promote quantified giving.

When rational and quantified giving are too strongly associated, rational giving suffers from the association. There are many legitimate criticisms of quantified giving that do not apply to rational giving, and future posts will advance some of these.

The primary purpose of this post was to draw as clear a distinction as we could. We don’t want to see the important and exciting project of “rational giving” remain tied to the much more limited and less exciting project of “quantified giving.”