We Can’t (Simply) Buy Capacity

Over the years, we’ve had many exchanges along these lines:

Q: Why can’t you [research more charities / research more causes / put more effort into marketing and outreach?]
A: We don’t have enough people; we’re already stretched thin with our current priorities.
Q: What if you had enough money to hire more people?
A: We have enough money to hire more people.
Q: Then why aren’t you hiring them?
A: We’re trying.
Q: But if you have the money, what’s the downside of hiring more people? Best case, you get more done. Worst case, you get the same amount done.

It seems many people imagine that hiring, and thus capacity building, is mostly a matter of trading money for staff-hours: if we don’t have enough people to move through our to-do list, but we do have enough money to hire more, we can quickly and smoothly expand the number of staff-hours available via hiring. We think this picture of the world has important shortcomings, especially when it comes to projects like GiveWell. In our view, hiring people – whether as employees or as third-party contractors – generally means sacrificing today’s staff capacity for future staff capacity, and is always a risky (though essential) long-term investment with a difficult-to-forecast payoff. (That doesn’t mean it isn’t worthwhile – quite the opposite, as we view GiveWell’s staff as absolutely central to what it has accomplished and will accomplish.) The less well-defined and easy-to-evaluate the role one is hiring for, the more this is true, and thus the less true it is that money is the only bottleneck to expansion.

This post lays out the thinking behind our view. It focuses on Elie’s and my experiences with recruiting and managing staff (though some of the staff we’ve hired have also participated, more recently, in recruiting and managing as well). We provide relatively little in the way of empirical support for the claims in this post because we can’t go into the details of individual staff performance, and when we’ve had conversations with outsiders about these issues we generally haven’t done conversation notes for them. For readers who wish to vet our claims, one possibility is to speak to people in your own network who have experience with startup organizations (as opposed to organizations with long-established and well-defined processes and outputs), and asking whether the claims in this post ring true to them.

In a nutshell:

  • In order to reach the point where employees are producing work that we can rely on and integrate into our output, senior staff need to invest significant time in training, evaluation and management.
  • Investing time in making better decisions in early stages of the selection process can save time in later stages. This can include investing time directly in evaluation as well as investing time in better understanding the nature of the work we’re assigning (so as to better understand how to evaluate it).
  • Hiring requires long-term predictions. There is often a substantial lag between hiring someone and having them start, and when an employee isn’t a good fit this can take time to determine. At the same time, the nature of GiveWell’s work is constantly changing. Thus, doing good evaluations of employees is both difficult and very important; the cost of an overly optimistic hire or evaluation can be significant.
  • These points are less true for work that is well-defined, straightforward to evaluate, and/or similar to other work people might have on their resumes. But most of the work we need done doesn’t fit this description.
  • Having adequate funds available is crucial, though not sufficient, for hiring. Cash on hand is less important than visibility into our long-term financial situation.

Growing our capacity has been a core priority of GiveWell for years. We believe it is extremely challenging and time-consuming for senior staff and often lowers rather than raises capacity in the short term, and we haven’t found a way around this fact. That said, it is also one of the most important and worthwhile things we do, and when done well can pay off massively over the long run. We feel that GiveWell’s success is ultimately mostly a function of how good a team we can build, and that GiveWell would be in a far worse position today if not for the outstanding people that have become integrated into the organization.

Employees need to be trained, evaluated and managed
In order for a new hire to save us time, it has to be the case that we rely on the new hire’s work in some sense. Research work that wasn’t careful enough, wasn’t thorough enough, or didn’t ask the questions we would ask couldn’t be used as a substantial input into our decisionmaking. Marketing work that represented GiveWell poorly could do more harm than good.

In the past, we’ve had significant problems come up when we gave an employee work and proceeded to rely on that work in some way, without first taking the time to vet the work, offer feedback, and form a thorough picture of the work’s alignment with our standards. We’ve learned to invest significantly – early in an employee’s tenure – in both training and evaluation, so that we have a clear picture of what to expect from any assignment we give, and so that we know what we can and can’t rely on the employee for given their level of ability and experience. We’ve also learned to do periodic in-depth evaluations of employees, as well as regular check-ins to discuss all aspects of their performance and assess how their responsibilities might change to find a better fit. There are some tasks that employees can do with fairly straightforward training, but there are other tasks that require deep and intuitive knowledge of GiveWell; so employees’ abilities often grow gradually and somewhat unpredictably, and constantly reassessing them pays off.

With these measures, we feel that we’re able to have a fairly good picture, at any given time, of each staff member’s strengths, weaknesses, and experience level, and this makes it possible to assign work that we’re able to rely on. Even so, as our needs and our employees’ abilities change, it can be a serious challenge to keep our picture accurate, and when our picture degrades, we often pay for it in the form of senior staff-hours (assessing the work and determining what went wrong and what adjustments can be made).

Predicting employee fit is difficult
We’ve found it to be very difficult to come to an accurate assessment of an employee’s strengths and weaknesses, particularly while recruiting/interviewing.

Traditional evaluation techniques – reading resumes, 1-2 hour interviews – have had extremely poor predictive value for us, as some of our outstanding hires have been “borderline” by such metrics while some “stars” by such metrics have turned out to be a poor fit. This has made it difficult to follow the traditional procedure of making a job posting and screening many resumes; instead, we have been trying to:

  • Improve the quality of the people we consider in the first place. We have dialed up our emphasis on referrals from trusted sources as opposed to job postings; we have also constructed our jobs page to discourage casual applications and select for people who are unusually passionate about the work we do. (We believe, though with only moderate confidence, that genuine passion for GiveWell’s work is correlated with strong job performance, as much of what we look for is the ability to ask the sorts of questions we would ask in an investigation.)
  • Assess people via trial hire periods whenever possible.

Exploring our network and assessing trial hires can cost significant staff time.

Even assessing an employee on the job can be very difficult. Because the nature of our work is constantly changing, we rarely have assignments that both (a) are relevant to our ongoing work; (b) can be performed without a great deal of experience on the job; (c) come with clear expectations and benchmarks in terms of what we can expect a strong new employee to produce. Without such assignments, we can’t form accurate assessments of employees and thus can’t make the most of the capacity we have. So it is sometimes the case that “lack of appropriate work to assign” is a bottleneck to hiring.

Sometimes, we believe the most productive way to make progress on hiring is to thoroughly – and personally – execute, explore and discuss the sort of work we hope to eventually outsource to employees. (This can involve multiple iterations as we find that a particular type of work turns out to be less relevant to our goals than we had anticipated.) When we thoroughly understand a type of work, we can be confident of its relevance to GiveWell, we can effectively train people on it, and we can accurately assess the quality of an employee’s work on it. We rarely try to assign work that we haven’t already done a fair amount of ourselves, and found to be relevant. Thus, hiring people to do work generally comes after – rather than instead of – doing it ourselves.

Hiring involves long-term decisions, and poor predictions can be costly
When we find someone we want to hire, it’s often the case that they aren’t available for a substantial amount of time. (Especially if they’re still in college, as is sometimes the case.)

In addition, we feel that hiring someone – especially when they forego other opportunities, quit their job, and/or move in order to join us – involves an implicit commitment to them, and if the relationship turns out to be a poor fit, we generally feel responsible for (a) clearly communicating that, and why, the relationship isn’t a fit; (b) doing what we can to explore whether there might be a way to improve the fit; (c) giving the employee sufficient advance notice to find another job.

Thus, the decision to hire someone often constitutes a risky “bet” that by the time they join GiveWell, their talents will still be a match for the work we need done; when we’re wrong, the process of determining this and ending the relationship can be difficult and costly. Making this sort of prediction is particularly difficult because of how quickly the nature of GiveWell’s work evolves. Starting research analysts do very different tasks today from what they did last year or the year before that.

In addition, we find all aspects of training, evaluation and management to be easier when we’re working in one office. This means that every employee affects the culture, environment, and morale of other employees. An employee who is dissatisfied or otherwise a poor fit can be a major problem.

Two pictures of hiring
The introduction of this post implies one view of hiring:

  1. Lay out an assignment that we need done.
  2. Post a job opportunity, collect resumes, and use a low-intensity process to evaluate candidates.
  3. Hire the best candidate to do the assignment.
  4. If it doesn’t go well, little is lost.

By contrast, the picture we have of hiring looks more like:

  1. Execute the work we need done ourselves, and form as deep an understanding of the work as we can, so that we can later train people in parts of it and evaluate their output. When exploring a new area of research, it can often take many false starts before we can generate assignments that are stably enough defined to assign to new employees.
  2. Create assignments that are both relevant to our work and relatively straightforward to evaluate.
  3. Use our network to find a promising candidate and do our best to assess their long-term fit with GiveWell based on the very limited information that we have.
  4. Work with the person on a trial basis, primarily aiming to train them and further assess their long-term fit with GiveWell. After each assignment, discuss what we’ve learned about the employee and how we can learn more, and communicate with the employee about future expectations.
  5. If all goes well, extend a full-time offer and continue to invest heavily in training, evaluation, and management.
  6. If there are problems, invest significant time in discussing and diagnosing the problems and determining whether the problems can be solved via communication with the employee or alteration of the employee’s role. If the nature of our work changes substantially (as happens regularly), make sure to revisit the employee’s fit with the work and ask what adjustments can be made.
  7. If the employee does well on early, easier-to-assess work, gradually expand the scope and challenge of the employee’s responsibilities while continuing to assess the employee’s work and train the employee.
  8. Ideally, the employee’s responsibilities and abilities gradually expand to the point where they become a major asset to the organization, capable of being relied on for crucial work, and in some cases capable of executing some or all of the above steps themselves (and thus increasing our “capacity for capacity building”).

The latter picture is one of risky, intensive long-time-horizon investments. There are cases in which hires or trial hires we’ve made have cost us more in training/evaluation/management time than they’ve gained us in work done, and we believe that there would be many more cases if we didn’t invest as much in early training and evaluation. However, the hires that have worked out well have eventually repaid the investment many times over, and GiveWell could not be what it is today without the strong staff we’ve built.

One of the most positive developments that can happen for GiveWell’s mission is that a hire develops to the point where we can trust them with highly challenging and important work (and that the employee stays with GiveWell for a substantial amount of time). Getting to that point requires major investment on our part in training, evaluation and management, so we pick the candidates for this carefully and invest in them heavily.

We haven’t found any shortcuts through this process (including, as discussed below, “outsourcing recruiting/management”). Building our capacity has been an extremely challenging and long-term project that has taken a great deal of co-Executive Director time and involved a great deal of learning.

When does the above dynamic hold and when doesn’t it?
There are certainly cases in which work can be outsourced without all of the challenges and management investment described above. For example, we’ve outsourced our bookkeeping and our back-end web development to contractors without nearly the challenges that we’ve encountered in hiring Research Analysts (though there have still definitely been challenges).

Generally, we’d say that it’s easier to “trade money for capacity” when:

  • The work we need done – and the expectations around what constitutes good work – is clearly and explicitly defined.
  • The work we need done is similar enough to work that is done elsewhere that we can, relatively easily, look at someone’s resume and credentials and assess their likelihood of being able to do it.

Bookkeeping services and back-end web development (of the relatively simple variety relevant to our website) are good examples of cases where we know what we need done and how to assess the output, and we know how to find people with relevant experience.

However, most of the work we need done is far to the other end of the spectrum: it is not well-defined, it’s hard for us to say what we’re looking for in terms of output, and it’s hard to predict from someone’s resume whether they will be a good fit for it. This work includes:

  • Research work including updates on top charities, investigations of potential top charities, intervention reports on things like cash transfers, conversations with experts, and shallow investigations for GiveWell Labs. While we’ve developed criteria and guidelines for this sort of work, it’s extremely far from being reduced to a formula, and anyone working on it will have to make substantial judgment calls in terms of which avenues of inquiry to pursue, which pieces of information to consider important, etc. We need to be sufficiently aligned with their judgment calls so that their work produces output that fits in with the rest of our work and our brand. We’ve experimented with recruiting people of multiple different backgrounds, and at this point we remain unaware of any particular resume item that would convince us that a person is a good fit for this role.
  • Communications/outreach/marketing. Communicating effectively about GiveWell requires a deep understanding of GiveWell as well as strong communications abilities. We have worked with communications/marketing professionals in the past and have always found that getting good output requires a significant amount of our own staff time. (They can be extremely helpful, but they can’t simply take on all of the work.)
  • Recruiting and management, including management of contractors (managing our bookkeeper, our web developer, etc.) We’ve toyed with the idea of trying to outsource management and recruiting, but the people we’ve spoken to about it – and our own intuitions – suggest that this would not go well. Management requires knowing our work and our needs well enough to be able to evaluate the quality of employees’ or contractors’ output; the bulk of recruiting work is evaluation (as opposed to sourcing) of candidates as well. Some of our Research Analysts have taken on management and recruiting work (including management of contractors), and in general we intend to develop Research Analysts into managers and recruiters.

The role of money
Expanding staff obviously requires money. In general, we believe that the people who are most likely to be a good fit are people who are genuinely passionate about our mission, and for this reason, we have consistently been able to hire these people for far less than what they could earn in e.g. the for-profit sector. However, we still need to be able to pay enough so that our employees are relatively comfortable and choose to stay on rather than looking for other work.

Hiring requires long-term visibility into our future financial situation, not just (and not even primarily) cash on hand. The effort we put into recruiting at a given time is a function of how much funding we think we’ll have available for the next several years, since the full process from “starting to look for potential hires” to “having a new hire start” can be 6-12 months or more, and since full-time hires are made with the hope that someone will be with us for several years at least. Observing this has helped us to recognize the value of consistent and predictable funding, and the complexity of assessing room for more funding for certain types of organizations.

We do our best to predict our ongoing revenue on the basis of what we know about our existing donors. A donation that we are confident will be repeating for the next several years is far more valuable to us than a donation whose likelihood of repeating we can’t predict.

While money is necessary for hiring, it is not sufficient. Simply being able to fundraise does not guarantee that we will be able to expand.

Conclusion
It seems to us that much of GiveWell’s future hinges on the effectiveness of senior staff in recruiting, selecting, training, evaluating and managing the best employees possible. If we could pick one set of skills to get better at, we would pick these. Investing in this goal – both directly (recruiting, interviewing, training, evaluating, managing) and indirectly (trying to understand our own work well enough to be able to outsource it) – takes up a very large proportion of our time, rightly so given the paramount importance of the goal.

We’ve tried many things, and talked to many people, over the years. We believe the best approach involves intensive up-front investment by senior staff, rather than low-touch management. Intensive up-front investment allows us to quickly form pictures of strengths and weaknesses and invest accordingly in training and development.

We don’t believe that money, alone, can solve this problem for us, though it is an essential piece of the picture.

Comments

We Can’t (Simply) Buy Capacity — 9 Comments

  1. I’ll briefly mention that, as the CEO of another “startup-style” non-university research institution with a comparable budget to GiveWell’s, I find that all of this rings true for me.

  2. Very interesting, thank you! One money-related matter you don’t touch on AFAICT is salary – to what extent would being able to offer a higher salary make recruiting easy for you? Are there people who you want and who want to work for you that can’t afford to accept what you’re currently able to offer?

  3. Paul: Think he’s saying, and I agree, that upping the salary beyond a certain point won’t help a ton, because good hires tend to want the job mostly because they really care about picking good charities. It certianly takes some amount to make it even possible for most folks to consider, but past some point increases could even be counterproductive: you might be attracting less-interested applicants who want the well-paying gig.

  4. I’m going to sound a bit ridiculous [edit: more ridiculous for the word count], but I’ve been surprised at the quality of content that communities of volunteers can collectively build up. I remember seeing Wikipedia early on and thinking the idea of starting a new encyclopedia from scratch without tons of funding was ridiculous. Even in the best case I don’t think a volunteer community could produce evaluations, but a large number of smart, motivated folks can at least surface a lot of sources and arguments, including on sources where GiveWell can’t spend a long time researching–that is, they don’t do your research assistant’s job, but they may be willing to provide your assistants a lot of materials and analysis of near-earth asteroid detection.

    You probably need certain things for it to work. You’d want participants who, assuming they’re making money at this point in their lives, donate something (no minimum, and no verification, but the norm would communicate that kibitzing without skin in the game is discouraged), and share some common premises (that cost-effectiveness and evidence matter, that all lives count as much as ones nearby, and so on). You want civility and citations to reliable sources; you want to avoid meaningless claims like you see in marketing materials, and otherwise you may want err towards acknowledging most any logically coherent argument. You need moderators who get those norms, and probably some trusted people authorized to kick out troublemakers. The motivation to come in is that you, and fellow donors, actually read this thing.

    Using Wikipedia as a model would make it seem too easy, because they have a ridiculous amount of raw traffic to work from and lots of long-time editors/administrators that clean up newbies’ messes. They also don’t take original research, and much of what GiveWell really needs *is* original research, as it isn’t that hard for Holden or someone to just read the stuff that’s already been cleanly written up.

    (On original research: someday, could volunteers eventually do stuff like get interviews with experts and charity officials, or could the occasional expert show up and contribute? Don’t know. I’ve seen volunteers do impressive stuff in political advocacy, but only after non-volunteers did a lot of work finding and helping them.)

    To a significant extent I think you’re seeking the benefits of open collaboration without making any big explicit push to build a forum for it, by inviting dissenting voices and comments, replying to other similarly-minded bloggers, and generally going for a transparent model where people can pick at, or augment, bits of your logic. But, anyway, part of me really wants to think that simply getting more input that meets a certain minimum standard, without all of the actors involved necessarily having reliably good judgment, is worth something.

  5. As an environmental consultant who has invested hundreds of hours in training brilliant young research assistants over the years, the biggest challenge for me isn’t so much finding the right people but rather keeping them. Most of our research assistants are recruited from university, and for many it’s their first fulltime job. After a few years many of them leave to go back to graduate school. Some of them come back afterward, but many don’t, and that training investment is lost. Sometimes I wonder if the answer isn’t to focus on hiring more mid-level and senior staff, who tend to be older and possibly more likely to stay longer and settle in. That wouldn’t work in my field, where cost-competitiveness is important and we thus need to rely heavily on junior staff in our projects, but it might in yours. It’s hard to maintain enthusiasm for training people when you know they’ll only be around for a few years and then you’ll have to train someone new.

  6. Paul and Randall, it’s certainly the case that some people require substantially more pay than others (based on different career stages, etc.) even when they buy heavily into the mission. So far, we’ve tried to stay open to the possibility of paying more if warranted, and thus I don’t believe that we have been held back along this dimension (though I can’t say for sure how we would have acted differently if we had had more “money moved” and/or operating support, and thus a clearer mandate for higher pay when warranted, in the past).

    Randall, I agree that there are cases in which volunteers can accomplish strikingly impressive things, but I also think there are many cases in which they can’t. We have had a lot of discussions around, and experiments involving, volunteers, and haven’t found a way to produce the kind of value you’re describing. Compared to Wikipedia, I think our work relies much more on contextual knowledge (understanding how many different things relate to each other as opposed to filling in isolated sets of facts) and is harder to create clear explicit standards for.

  7. Good to know you guys have at least walked around the notion of taking more advantage of volunteers, and I do agree GiveWell’s task presents some particular problems for trying to integrate volunteer work (and that the problems may even be insurmountable).

  8. It sounds like a lot of the problem you have surrounding recruiting and retention are relation to the fact that “the nature of GiveWell’s work is constantly changing”. Regarding this issue:
    1. Do you expect this to continue to be the case indefinitely?
    2. Is it possible to predict to some extent what changes might happen in the future, so as to allow for a longer recruiting pipeline?
    3. What are the source of these changes, and would it be possible/desirable to mitigate them?

  9. Ian:

    1. I’m not sure. I think our “traditional” (proven, cost-effective, scalable) work changes less quickly today than it used to and is approaching stability. It’s hard to predict when, or whether, or work on Labs will follow the same pattern.

    2. It’s not. What we can do is get better at predicting which skills are most robustly useful at GiveWell even as the nature of GiveWell’s work changes. For example, one thing we’ve found over time is that an employee’s propensity to “check in” frequently, so that high-level decisions about the work they’re doing can be made collaboratively, seems to predict the employee’s performance well on a wide variety of tasks.

    3. GiveWell was founded to answer a very broad and challenging question (where should I give?) as efficiently as possible, and I think it’s a good thing that we constantly evolve our approach and change our minds about the best processes for doing so. We don’t, today, know for sure what the best research process to follow is to make progress on this question, and we intend to continue to evolve the process rather than stick to a suboptimal formula. Over time we have seen, and hopefully will continue to see, progress on identifying robustly useful sub-tasks. But I don’t think we would be well served by forcing our process to be more stable.