The Origins and Future of Scrum: An Interview with Ken Schwaber

Charles Suscheck: Ken, you've been in Scrum for long time; well over ten years. It's a really different way of looking at software development. What triggered your mind, in those early days, to shift from a predictive approach to an empirical approach?

Ken Schwaber: I've been working with the ideas of Scrum for twenty-three years. Back then, we were building the first ALM tools in Burlington, MA, but the tools weren’t working very well. They were online and they weren't adopted. So we looked at a way to connect the methodology to the tools so that they would work better.

It was great, except I was using one of the early OO technologies and, at the same time, the customer was coming back with changes—new built-in capabilities or customizations. We were going nuts. The ability to deal both with the technology difficulties and the constant requests for changes lead me to start coming up with the ideas in Scrum and agile. At the same time, Jeff Sutherland, who is a friend of mine, was running into the same problems.

Turns out, we were trying to solve the same problems, so we started collaborating and found out we were doing pretty much the same thing. Jeff was formalizing these ideas from Scrum, so we started doing that together. We worked some more, and I started using the Scrum terminology as we fleshed it out. By 1995, we had a pretty well-described process that we called Scrum, which had a ScrumMaster and product owner and many of the current ideas.

The thing that drove me to these ideas was trying to be responsive to many customers and keep them in the loop. We would have gone out of business if we didn't work that way.

As Jeff and I were formalizing Scrum, we got an email from Kent Beck, who had seen our initial work at OOPSLA. He said those are really neat ideas and asked to borrow some—that is why you see convergence between XP and Scrum. I had always hoped, particularly leading up to the 2001 Agile Manifesto, that we could somehow band together, because Scrum answers the question of how to develop within an agile framework.

Charles: I read a lot of studies that say the adaptation of Scrum in agile organizations is well over 84 percent. Many companies are using Scrum as their primary framework. Does that surprise you?

Ken: Well, the latest data from Forester says 92 percent of agile organizations are using Scrum. I would have expected at this point to see 90 to 92 percent Scrum and maybe 85 percent XP. So it doesn't surprise me, because I think the statistics are a little funny. The data doesn't say how many organizations are developing software just with Scrum and how many are using modern engineering techniques, too. It just asks if you are using Scrum in your organization. I think if we ask the other question, the answer might be closer to 30 or 40 percent, which is still pretty reasonable. The most pleasurable thing is I think we have finally turned a corner on people thinking waterfall is a good approach for software development. As long as that type of mentality persisted, there was always the chance that we could have something like RUP or waterfall again.

Charles: What do you think are some of the big threats to Scrum adoption or growth?

Ken: I think the biggest threat is people's desire for certainty and their belief that somehow, if they can nail things down just a little better, they will have certainty. You see that with Safe and DAD and even the desire for kanban, which has more certainty than Scrum. I see these edges being eaten away by people wanting more certainty. Even within Scrum there are conversations about how to get more predictability of velocity, and the effort that people put into the sprint plan meeting trying to be more certain of what we are going to deliver at the end of the sprint. All that is absolutely contrary from the biggest idea in Scrum: Let's use short cycles so we don't care about how much we create, because we are going to find out very soon.

I heard a year ago at an ALM conference that one guy was giving a talk saying agile was a fad and that software has a fad every ten years. He was talking about what the next biggest fad would be. That scared me, because I think the basic ideas are shifting the way of thinking in Scrum toward empiricism and complexity. If they view that as a fad, then it can swing back to “Let's try to control this” and that's dangerous.

What I see happening over the past three years is that everyone and their brother is coming up with a new way to make money off of agile. Like the CMMI's idea to warrant software was ruined by turning a good set of ideas into just a way of making money. I see all these things that are not really thought through but just an application of old thinking to the agile movement, and they stand a good chance of crushing agile; of turning it into a nice set of ideas that are honored in word but not in principle.

Charles: What do you see as the next big thing coming with the Scrum framework?

Ken: For us, I think the next big thing is evidence-based management (EBM)—showing the value of doing work one way or another. Not only for businesses to see the value but, if they don't act on the value, it should be clear that they are doing things that are not beneficial to their corporation.

EBM just says "Here's some metrics we can use to see how we are doing," and these are things that I could take right into management and show them. It is appealing because it doesn't even start with the idea of change. It simply starts with the thought that it's really a good idea to try to measure what is going on.

It’s been a problem that management only sees IT as an expense that doesn't provide much value. If we measure business value, we suddenly change the way we structure with upper management. They think "This interesting, why don't we measure this in terms of value and see how much value is actually being provided by IT." Now, of course, you can't do that in the absolute sense, you can't say "Oh, this is 17.4 oz of value," but you can start setting baselines so you can compare trends and predict based on your actions what will happen over time.

Another problem is that agility has not been measured. Jeff and I have seen over and over again a company getting really, really good at agile and their management not believing it is going to do any good. The idea in EBM is that if you have measurements, just like scales have measurements and manufacturing has measurements, you can see the effects of change and agile.

You are bringing management a new way to tell if they are continuing to get better or worse. I'm hoping that between two thrusts this will work. One thrust is management hasn't been measured in IT or product development. Ever. They have been under pressure to have a way of measuring so they can report to their CEO, and EBM gives them an answer.

The second thrust is that the measurements that they use spotlight their actions and hold management accountable for some degree of performance in the same way the rest of the organization is. This is a really sneaky thing that raises transparency at the management level just like we have at the development level.

Charles: It sounds like EBM gives freedom to the managers as well as accountability. Freedom in that they don't have to rely on their own decisions and their own intuition, but accountability in that they now know what they can be measured against and that it's something that's not just a mechanical measure.

Ken: Yes. Exactly.

I think some managers are not going to welcome this approach because they are only responsible for explaining why projects didn't deliver on time and they are able to blame someone else. This is going to be really, really scary to a good number of managers who aren't very confident in what they do.

I think that when they all start looking at the metrics and start seeing what is really happening, that this is going to be pretty interesting. One of the sneaky things in EBM and the agility path is the agility index. This is the earliest metric to compare as a trend within your organization (that's pretty scary right there), but we can gather more data for comparisons between like organizations in the same industry sector. All you need to do is put out a report with inter-corporate metrics and have it show up in Forester, Gartner, and other newspapers, and you'll start seeing a lot of interesting discussions.

The idea is appealing; however, once you start using it, it starts coming back on you and you have to do something to improve the organization. And for EBM, that's the hook; that's the trick that's in it.

Ken Schwaber co-developed the Scrum process with Jeff Sutherland in the early 1990s to help organizations struggling with complex development projects. Ken is founder of Scrum.org and one of the signatories to the Agile Manifesto in 2001. He subsequently founded the Agile Alliance and Scrum Alliance. He recently left the Scrum Alliance to found Scrum.org.

A thirty-year veteran of the software development industry (from bottle washer to boss), Ken has written three books about Scrum: Agile Software Development with Scrum, Agile Project Management with Scrum, and The Enterprise and Scrum.