I previously wrote about the challenges of capacity building – hiring, training, and managing a team. We thought we would share some of the principles and practices we’ve come to believe are important to this goal, so that others can both learn from/consider them and provide their thoughts. As in the previous post on this topic, “we” primarily means Elie and myself, as it’s only fairly recently that other staff have taken on managerial roles.

An overriding theme is that of putting substantial work into capacity building (training, evaluating, and managing employees) and constantly integrating capacity-building considerations into our everyday work. In every piece of work we do and every assignment we make, we ask whether the way we’re going about it is optimized for long-term capacity building (training and evaluating other staff such that they end up in the best possible roles), and not just for getting the work done as quickly as possible. Maintaining this attitude can be a challenge, since it involves focusing on a longer-term (and hazier) goal than completion of the work at hand, but we believe that it is the right long-run approach to building a team.

• We try to think constantly about whether everyone, including and especially ourselves, is assigned work that’s a good fit. Whenever I start a task, I ask, “Am I the best person to do this task? Can someone else on staff, who has capacity, do it just as well? Why does it have to be me?”
• When we don’t have much sense of what we’re expecting from an assignment, we’re likely to take it on ourselves. By doing the work ourselves, we’re able to get a better picture of how relevant it is and what it consists of, which is necessary if we’re hoping to later assign similar work to more junior staff. (We do have a “Senior Research Analyst” distinction for other staff members that we’re comfortable assigning extremely open-ended work; currently we have one Senior Research Analyst, Alexander Berger.)
• When we’re doing work ourselves, we often explicitly approach the work with a goal of “getting the work to the point where other staff can carry it on” rather than with a goal of “completing the work as efficiently as possible.” This means a lot of reflection and (sometimes) writing about what we’ve been doing, and thinking about what skills and knowledge base it requires. For example, Elie recently completed a shallow investigation of developing-world infrastructure that we explicitly thought of as a way to reflect on how shallow investigations should be done in general; I am currently focused on learning enough about political advocacy and scientific research to create shallow investigation assignments in these areas for other staff. In general, we see much of our GiveWell Labs work as “learning how to do investigations (so that we can build staff capacity for them via hiring and training)” rather than as “simply trying to understand the topics we’re exploring.”
• When we assign work, we usually are optimizing for the goal of “training and evaluating staff” rather than simply “completing the work as efficiently as possible.” Ideally, we want each assignment to help the assigned employee learn, and to help us learn about the employee. We think about how each assignment fits into our picture of the employee’s strengths, weaknesses, areas for improvement, and possible long-term trajectory. Sometimes the primary purpose of an assignment is to evaluate an employee, including “stretch” work that gives the employee an opportunity to show that they can take on more responsibility than they have in the past.
• We try to have a reasonably clear picture of what constitutes “excellent,” “good” and “subpar” work so we can make accurate evaluations. Doing so can be very difficult, and is usually more tractable when we have more experience with a given assignment.
• When we have an assignment that’s particularly easy to evaluate, we’ll sometimes assign it to multiple employees over time for calibration purposes (i.e., getting a read on different employees’ strengths and weaknesses).
• When assigning work that is very different from what an employee has done before, we often follow the work very closely, and we are often prepared to spend more time managing the assignment than it would have taken to complete the assignment ourselves. We seek to thoroughly understand, and give our input into, the employee’s thought process and progress, so that the output is a meaningful representation of what will be produced on future similar assignments. We’ve found that a major determinant of an employee’s fit with GiveWell is their willingness to constantly check in with us and keep us posted on their thought process, so that we can stay on the same page.
• When we gain confidence in an employee’s ability to do a particular kind of work, we step back and reduce our involvement – but we still take periodic opportunities to check in at a greater level of depth. Occasionally Elie or I will “double-do” an employee’s work to see whether we pick up any new insights about how the work is being done and how it could be improved. Most conversations at GiveWell, external and internal, are recorded, and Elie has found it helpful to listen to them when he can.
• We put significant time and reflection into considering employees’ long-term trajectories and try to make assignments consistent with these.
• We spend significant time in discussions with employees, getting on the same page about both strengths and weaknesses of recent assignments and likely long-term trajectories. It’s important that employees know where we stand and have the opportunity to point out when they disagree with us on these matters.

We’d welcome thoughts from others on these principles and practices.

This is just a reminder to donors considering supporting our top charities in the next few months. As we do every year, we’re planning to refresh our top charity rankings before December 1st.

Over the past few weeks, we’ve seen something that we see too rarely: a debate over philanthropy.

Bill Schambra, at the invitation of the Hewlett Foundation (a funder of GiveWell), attacked the idea of “strategic philanthropy” presented in Money Well Spent (a book co-authored by Paul Brest, former Hewlett President). He characterized Money Well Spent as promoting a world in which nonprofits “sit around drinking herbal tea … filling whiteboards with guesses about why people have needs, and how to avert them altogether by getting at their root causes … [putting] more trust in the college degrees earned by the public health director than in our hard-won, practical, face-to-face understanding of our own community.” Critiques of Mr. Schambra’s argument have been posted by Paul Brest, Larry Kramer (the Hewlett Foundation’s current President) and Phil Buchanan (President of the Center for Effective Philanthropy).

Because we so rarely witness philanthropists publicly arguing with each other about how to do good philanthropy, we’re glad to see that Hewlett encouraged Mr. Schambra to put forth a critique and engaged it directly. However, we think there is a major and extremely telling missing element from this debate. The case study being debated is entirely hypothetical, involving an investigation by the “hypothetical Metro Community Foundation” of the reasons for a winter-time surge in demand for food and other necessities.

In our view, this is a sobering (and accurate) reflection on the state of foundation transparency, as well as the state of historical study of philanthropy.

Every day, people have public debates about how we should be governed, what companies we should invest in, and a wide variety of other topics, with a broad set of news and facts to draw on in challenging each other. But when it comes to how we should give, not only are there few debates, but the debates that do occur have very little to work with in terms of substantive information provided by foundations and/or independent investigators. It isn’t that there is no alternative to using hypotheticals (Money Well Spent discusses actual cases as well, and hypotheticals can have advantages in conceptual discussions); but the relative paucity of concrete examples, informative case studies, etc. is striking.

We believe that a change along this dimension would be greatly beneficial for people’s abilities to make good giving decisions, which is why we are investing in both transparency and historical study of philanthropy.

 

 

Over the years, we’ve had many exchanges along these lines:

Q: Why can’t you [research more charities / research more causes / put more effort into marketing and outreach?]
A: We don’t have enough people; we’re already stretched thin with our current priorities.
Q: What if you had enough money to hire more people?
A: We have enough money to hire more people.
Q: Then why aren’t you hiring them?
A: We’re trying.
Q: But if you have the money, what’s the downside of hiring more people? Best case, you get more done. Worst case, you get the same amount done.

It seems many people imagine that hiring, and thus capacity building, is mostly a matter of trading money for staff-hours: if we don’t have enough people to move through our to-do list, but we do have enough money to hire more, we can quickly and smoothly expand the number of staff-hours available via hiring. We think this picture of the world has important shortcomings, especially when it comes to projects like GiveWell. In our view, hiring people – whether as employees or as third-party contractors – generally means sacrificing today’s staff capacity for future staff capacity, and is always a risky (though essential) long-term investment with a difficult-to-forecast payoff. (That doesn’t mean it isn’t worthwhile – quite the opposite, as we view GiveWell’s staff as absolutely central to what it has accomplished and will accomplish.) The less well-defined and easy-to-evaluate the role one is hiring for, the more this is true, and thus the less true it is that money is the only bottleneck to expansion.

This post lays out the thinking behind our view. It focuses on Elie’s and my experiences with recruiting and managing staff (though some of the staff we’ve hired have also participated, more recently, in recruiting and managing as well). We provide relatively little in the way of empirical support for the claims in this post because we can’t go into the details of individual staff performance, and when we’ve had conversations with outsiders about these issues we generally haven’t done conversation notes for them. For readers who wish to vet our claims, one possibility is to speak to people in your own network who have experience with startup organizations (as opposed to organizations with long-established and well-defined processes and outputs), and asking whether the claims in this post ring true to them.

In a nutshell:

• In order to reach the point where employees are producing work that we can rely on and integrate into our output, senior staff need to invest significant time in training, evaluation and management.
• Investing time in making better decisions in early stages of the selection process can save time in later stages. This can include investing time directly in evaluation as well as investing time in better understanding the nature of the work we’re assigning (so as to better understand how to evaluate it).
• Hiring requires long-term predictions. There is often a substantial lag between hiring someone and having them start, and when an employee isn’t a good fit this can take time to determine. At the same time, the nature of GiveWell’s work is constantly changing. Thus, doing good evaluations of employees is both difficult and very important; the cost of an overly optimistic hire or evaluation can be significant.
• These points are less true for work that is well-defined, straightforward to evaluate, and/or similar to other work people might have on their resumes. But most of the work we need done doesn’t fit this description.
• Having adequate funds available is crucial, though not sufficient, for hiring. Cash on hand is less important than visibility into our long-term financial situation.

Growing our capacity has been a core priority of GiveWell for years. We believe it is extremely challenging and time-consuming for senior staff and often lowers rather than raises capacity in the short term, and we haven’t found a way around this fact. That said, it is also one of the most important and worthwhile things we do, and when done well can pay off massively over the long run. We feel that GiveWell’s success is ultimately mostly a function of how good a team we can build, and that GiveWell would be in a far worse position today if not for the outstanding people that have become integrated into the organization.

Employees need to be trained, evaluated and managedIn order for a new hire to save us time, it has to be the case that we rely on the new hire’s work in some sense. Research work that wasn’t careful enough, wasn’t thorough enough, or didn’t ask the questions we would ask couldn’t be used as a substantial input into our decisionmaking. Marketing work that represented GiveWell poorly could do more harm than good.

In the past, we’ve had significant problems come up when we gave an employee work and proceeded to rely on that work in some way, without first taking the time to vet the work, offer feedback, and form a thorough picture of the work’s alignment with our standards. We’ve learned to invest significantly – early in an employee’s tenure – in both training and evaluation, so that we have a clear picture of what to expect from any assignment we give, and so that we know what we can and can’t rely on the employee for given their level of ability and experience. We’ve also learned to do periodic in-depth evaluations of employees, as well as regular check-ins to discuss all aspects of their performance and assess how their responsibilities might change to find a better fit. There are some tasks that employees can do with fairly straightforward training, but there are other tasks that require deep and intuitive knowledge of GiveWell; so employees’ abilities often grow gradually and somewhat unpredictably, and constantly reassessing them pays off.

With these measures, we feel that we’re able to have a fairly good picture, at any given time, of each staff member’s strengths, weaknesses, and experience level, and this makes it possible to assign work that we’re able to rely on. Even so, as our needs and our employees’ abilities change, it can be a serious challenge to keep our picture accurate, and when our picture degrades, we often pay for it in the form of senior staff-hours (assessing the work and determining what went wrong and what adjustments can be made).

Predicting employee fit is difficultWe’ve found it to be very difficult to come to an accurate assessment of an employee’s strengths and weaknesses, particularly while recruiting/interviewing.

Traditional evaluation techniques – reading resumes, 1-2 hour interviews – have had extremely poor predictive value for us, as some of our outstanding hires have been “borderline” by such metrics while some “stars” by such metrics have turned out to be a poor fit. This has made it difficult to follow the traditional procedure of making a job posting and screening many resumes; instead, we have been trying to:

• Improve the quality of the people we consider in the first place. We have dialed up our emphasis on referrals from trusted sources as opposed to job postings; we have also constructed our jobs page to discourage casual applications and select for people who are unusually passionate about the work we do. (We believe, though with only moderate confidence, that genuine passion for GiveWell’s work is correlated with strong job performance, as much of what we look for is the ability to ask the sorts of questions we would ask in an investigation.)
• Assess people via trial hire periods whenever possible.

Exploring our network and assessing trial hires can cost significant staff time.

Even assessing an employee on the job can be very difficult. Because the nature of our work is constantly changing, we rarely have assignments that both (a) are relevant to our ongoing work; (b) can be performed without a great deal of experience on the job; (c) come with clear expectations and benchmarks in terms of what we can expect a strong new employee to produce. Without such assignments, we can’t form accurate assessments of employees and thus can’t make the most of the capacity we have. So it is sometimes the case that “lack of appropriate work to assign” is a bottleneck to hiring.

Sometimes, we believe the most productive way to make progress on hiring is to thoroughly – and personally – execute, explore and discuss the sort of work we hope to eventually outsource to employees. (This can involve multiple iterations as we find that a particular type of work turns out to be less relevant to our goals than we had anticipated.) When we thoroughly understand a type of work, we can be confident of its relevance to GiveWell, we can effectively train people on it, and we can accurately assess the quality of an employee’s work on it. We rarely try to assign work that we haven’t already done a fair amount of ourselves, and found to be relevant. Thus, hiring people to do work generally comes after – rather than instead of – doing it ourselves.

Hiring involves long-term decisions, and poor predictions can be costlyWhen we find someone we want to hire, it’s often the case that they aren’t available for a substantial amount of time. (Especially if they’re still in college, as is sometimes the case.)

In addition, we feel that hiring someone – especially when they forego other opportunities, quit their job, and/or move in order to join us – involves an implicit commitment to them, and if the relationship turns out to be a poor fit, we generally feel responsible for (a) clearly communicating that, and why, the relationship isn’t a fit; (b) doing what we can to explore whether there might be a way to improve the fit; (c) giving the employee sufficient advance notice to find another job.

Thus, the decision to hire someone often constitutes a risky “bet” that by the time they join GiveWell, their talents will still be a match for the work we need done; when we’re wrong, the process of determining this and ending the relationship can be difficult and costly. Making this sort of prediction is particularly difficult because of how quickly the nature of GiveWell’s work evolves. Starting research analysts do very different tasks today from what they did last year or the year before that.

In addition, we find all aspects of training, evaluation and management to be easier when we’re working in one office. This means that every employee affects the culture, environment, and morale of other employees. An employee who is dissatisfied or otherwise a poor fit can be a major problem.

Two pictures of hiringThe introduction of this post implies one view of hiring:

1. Lay out an assignment that we need done.
2. Post a job opportunity, collect resumes, and use a low-intensity process to evaluate candidates.
3. Hire the best candidate to do the assignment.
4. If it doesn’t go well, little is lost.

By contrast, the picture we have of hiring looks more like:

1. Execute the work we need done ourselves, and form as deep an understanding of the work as we can, so that we can later train people in parts of it and evaluate their output. When exploring a new area of research, it can often take many false starts before we can generate assignments that are stably enough defined to assign to new employees.
2. Create assignments that are both relevant to our work and relatively straightforward to evaluate.
3. Use our network to find a promising candidate and do our best to assess their long-term fit with GiveWell based on the very limited information that we have.
4. Work with the person on a trial basis, primarily aiming to train them and further assess their long-term fit with GiveWell. After each assignment, discuss what we’ve learned about the employee and how we can learn more, and communicate with the employee about future expectations.
5. If all goes well, extend a full-time offer and continue to invest heavily in training, evaluation, and management.
6. If there are problems, invest significant time in discussing and diagnosing the problems and determining whether the problems can be solved via communication with the employee or alteration of the employee’s role. If the nature of our work changes substantially (as happens regularly), make sure to revisit the employee’s fit with the work and ask what adjustments can be made.
7. If the employee does well on early, easier-to-assess work, gradually expand the scope and challenge of the employee’s responsibilities while continuing to assess the employee’s work and train the employee.
8. Ideally, the employee’s responsibilities and abilities gradually expand to the point where they become a major asset to the organization, capable of being relied on for crucial work, and in some cases capable of executing some or all of the above steps themselves (and thus increasing our “capacity for capacity building”).

The latter picture is one of risky, intensive long-time-horizon investments. There are cases in which hires or trial hires we’ve made have cost us more in training/evaluation/management time than they’ve gained us in work done, and we believe that there would be many more cases if we didn’t invest as much in early training and evaluation. However, the hires that have worked out well have eventually repaid the investment many times over, and GiveWell could not be what it is today without the strong staff we’ve built.

One of the most positive developments that can happen for GiveWell’s mission is that a hire develops to the point where we can trust them with highly challenging and important work (and that the employee stays with GiveWell for a substantial amount of time). Getting to that point requires major investment on our part in training, evaluation and management, so we pick the candidates for this carefully and invest in them heavily.

We haven’t found any shortcuts through this process (including, as discussed below, “outsourcing recruiting/management”). Building our capacity has been an extremely challenging and long-term project that has taken a great deal of co-Executive Director time and involved a great deal of learning.

When does the above dynamic hold and when doesn’t it?There are certainly cases in which work can be outsourced without all of the challenges and management investment described above. For example, we’ve outsourced our bookkeeping and our back-end web development to contractors without nearly the challenges that we’ve encountered in hiring Research Analysts (though there have still definitely been challenges).

Generally, we’d say that it’s easier to “trade money for capacity” when:

• The work we need done – and the expectations around what constitutes good work – is clearly and explicitly defined.
• The work we need done is similar enough to work that is done elsewhere that we can, relatively easily, look at someone’s resume and credentials and assess their likelihood of being able to do it.

Bookkeeping services and back-end web development (of the relatively simple variety relevant to our website) are good examples of cases where we know what we need done and how to assess the output, and we know how to find people with relevant experience.

However, most of the work we need done is far to the other end of the spectrum: it is not well-defined, it’s hard for us to say what we’re looking for in terms of output, and it’s hard to predict from someone’s resume whether they will be a good fit for it. This work includes:

• Research work including updates on top charities, investigations of potential top charities, intervention reports on things like cash transfers, conversations with experts, and shallow investigations for GiveWell Labs. While we’ve developed criteria and guidelines for this sort of work, it’s extremely far from being reduced to a formula, and anyone working on it will have to make substantial judgment calls in terms of which avenues of inquiry to pursue, which pieces of information to consider important, etc. We need to be sufficiently aligned with their judgment calls so that their work produces output that fits in with the rest of our work and our brand. We’ve experimented with recruiting people of multiple different backgrounds, and at this point we remain unaware of any particular resume item that would convince us that a person is a good fit for this role.
• Communications/outreach/marketing. Communicating effectively about GiveWell requires a deep understanding of GiveWell as well as strong communications abilities. We have worked with communications/marketing professionals in the past and have always found that getting good output requires a significant amount of our own staff time. (They can be extremely helpful, but they can’t simply take on all of the work.)
• Recruiting and management, including management of contractors (managing our bookkeeper, our web developer, etc.) We’ve toyed with the idea of trying to outsource management and recruiting, but the people we’ve spoken to about it – and our own intuitions – suggest that this would not go well. Management requires knowing our work and our needs well enough to be able to evaluate the quality of employees’ or contractors’ output; the bulk of recruiting work is evaluation (as opposed to sourcing) of candidates as well. Some of our Research Analysts have taken on management and recruiting work (including management of contractors), and in general we intend to develop Research Analysts into managers and recruiters.

The role of moneyExpanding staff obviously requires money. In general, we believe that the people who are most likely to be a good fit are people who are genuinely passionate about our mission, and for this reason, we have consistently been able to hire these people for far less than what they could earn in e.g. the for-profit sector. However, we still need to be able to pay enough so that our employees are relatively comfortable and choose to stay on rather than looking for other work.

Hiring requires long-term visibility into our future financial situation, not just (and not even primarily) cash on hand. The effort we put into recruiting at a given time is a function of how much funding we think we’ll have available for the next several years, since the full process from “starting to look for potential hires” to “having a new hire start” can be 6-12 months or more, and since full-time hires are made with the hope that someone will be with us for several years at least. Observing this has helped us to recognize the value of consistent and predictable funding, and the complexity of assessing room for more funding for certain types of organizations.

We do our best to predict our ongoing revenue on the basis of what we know about our existing donors. A donation that we are confident will be repeating for the next several years is far more valuable to us than a donation whose likelihood of repeating we can’t predict.

While money is necessary for hiring, it is not sufficient. Simply being able to fundraise does not guarantee that we will be able to expand.

ConclusionIt seems to us that much of GiveWell’s future hinges on the effectiveness of senior staff in recruiting, selecting, training, evaluating and managing the best employees possible. If we could pick one set of skills to get better at, we would pick these. Investing in this goal – both directly (recruiting, interviewing, training, evaluating, managing) and indirectly (trying to understand our own work well enough to be able to outsource it) – takes up a very large proportion of our time, rightly so given the paramount importance of the goal.

We’ve tried many things, and talked to many people, over the years. We believe the best approach involves intensive up-front investment by senior staff, rather than low-touch management. Intensive up-front investment allows us to quickly form pictures of strengths and weaknesses and invest accordingly in training and development.

We don’t believe that money, alone, can solve this problem for us, though it is an essential piece of the picture.