The Great Agile and Waterfall Debate: An Interview with Calvin Baldwin

Cameron Philipp-Edmonds: All right, today we are joined by Calvin Baldwin, who's going to be speaking to us today about agile versus waterfall. First, let me say thank you for joining us today.

Calvin Baldwin: You're welcome. I'm excited.

Cameron: Thank you. Tell us a little bit about yourself and your current role.

Calvin: Well, about me: I guess I would consider myself a thought leader. I've always been a person who challenges norms and tries to embrace change even when it's least desired. As for my career: I started my career as an independent consultant, after spending over a decade at IBM. However, I’m most proud of my training at top ranked engineering and business schools. This foundation prepared me to consult at various Fortune 100 companies and to lead in fast-paced corporate environments.

In my current role, I work with growth e-commerce companies as a software management consultant. I specialize in software delivery. I coach c-level managers and technical teams as they deploy leading edge software solutions to digital properties. I help technical and business organizations increase productivity, reduce cost, and increase customer satisfaction; as they scale to meet increasing customer demand.

Cameron: Now you recently wrote an article that was featured on LinkedIn titled "Agile Versus Waterfall" and for anyone out there that hasn't read it yet, I encourage them to do so. It is part one of a two-part exploration into the differences between the two methodologies. What led you to writing these articles?

Calvin: Well, it really came from a passion. I enjoy software change management, working with growing businesses, implementing new solutions, and solving the big challenges companies face with software delivery. After many years of consulting, I’ve had the opportunity to experience the hurdles and roadblocks companies face while growing. I’ve seen the good, the bad, and the ugly of agile and waterfall implementations.

One day while sitting around feeling the need to be creative, it occurred to me that I could help more companies and people through writing. I reflected on how agile is growing fast but many companies are struggling with how to add it to their organizations. I thought about several executives that I’ve consulted with and their desire to find better ways to produce high quality software on budget and in-time to meet increasing customer demand.

Through writing, I believe I can help a larger audience of people who are in the process of considering agile or waterfall. Mainly for those who are seriously trying to figure out what best fits their organization. I can coach and mentor others by sharing lessons learned with the audience on LinkedIn. 

Cameron: In agile versus waterfall, it's such a hot debate and such a hot topic and has been for the last ten years. Have you thought about expanding out past two parts of the series?

Calvin: Yeah, and I'm glad you asked. The first article has been so successful that I have extended it to a series. Right now, I have about seven additional articles planned to cover the topic. I've been getting great feedback from other thought leaders and subject-matter experts. They find my approach different because I don’t pick a side and state which is better or worse. Instead they appreciate that I go deeper. I’m excited because I love this stuff! I will, definitely, continue to write more on the topic.

Cameron: All right, fantastic. Now to cover things up a little bit with some housekeeping first: In brief, could you tell us what the waterfall method is, and who typically uses this method of project management?

Calvin: Waterfall project management (a sequential software development methodology) is what I consider a traditional style of project management that's been around for over five generations. The significance of this is that it was not created for the software industry. The enterprise software industry is fairly young with Google being two generations old and Facebook one generation old. None-the-less waterfall, has been very well received and there's tons of organizations that certify and back waterfall as a best practice approach to project management. Waterfall’s numerous endorsements has helped it become widely accepted. Typically, technology organizations older than fifteen years use waterfall.

Cameron: On the flip side of the coin, there's the agile methodology. In your words, what is agile, and who should be interested in adopting this method?

Calvin: Agile is the more contemporary of the two methods and a little bit older than one generation. If waterfall's the incumbent, then the new candidate would definitely be agile. It focuses on improving how project teams work together and accomplish goals. I consider agile a software management philosophy, its ideology focuses on the people component of software management. Agile is a collection of lessons learned, a list of concepts compiled which allow software to be done better.

One thing that makes agile significant is how it originated. It essentially was created by an ad-hoc team of seventeen thought leaders on software delivery focused on doing software better and helping others with this goal. I personally consider this approach, and the philosophy of agile, to be focused on the people instead of the process and the procedures.

Agile works well in fast-paced collaborative environments. You asked who should be using it. Any manager looking to improve the speed and quality of software delivered should consider agile. However it’s not for all organizations. In part two of agile vs. waterfall I answer this question with detail, I show managers how to use systematic analysis to determine if agile is right for their organization.

Cameron: And now that you've explained it, they kind of seem relatively similar, and they have the same end, yet many people are very passionate about one side of the other, and it is a hot debate. What are the fundamental differences about the two that puts them at odds with each other? Why can't they co-exist?

Calvin: That's a great question. I would have to highlight the fast changing customer demand but most importantly the people component. Agile focuses on the communication, the team, and the people. Agile practitioners state, "We need to produce quality software. We need to produce it fast. And we need the latest and greatest requirements, not requirements of two months ago." Waterfall is more of a top-down business operations process where practitioners ask, "How do you get this done, save money and reduce risk?" 

Waterfall was created many generations ago, it originated when manufacturing was driving the economy not technology. However, agile originated little over a decade ago specifically for software development and during a time of exponential enterprise software growth. This is a big change in how work is completed.

The models are at odds due to people. A person can understand the need for change but still provide resistance when they don’t know what this change means for them personally. For example, I recently chatted with a software business analyst (BA), he was very upset and uncomfortable in his agile BA role. He simply didn’t feel that he fit-in with his new agile software team. Why? Well, he had spent more than fifteen years working as a waterfall BA. As a result, he was accustomed to spending two to three months developing detail requirement documents that articulated a client’s expectations with precision. However with his new agile team he was expected to produce agile requirements in one week. This was a different way of working; this person was not ready or willing to adjust to learn agile requirements management. Although, this is a very specific example I see similar issues often with individuals and management teams. 

Agile technical teams work closely with the client on a day-to-day basis as they complete small chunks of work. While waterfall allows individuals to work separately, with different routines, as they complete large chunks of work. On a waterfall project, the client/business team is likely to create a requirements document three months ago, throw it over the fence to technology team and say, "Hey, we want this product completed in two and a half months." These actions can frustrate any technical team. 

When it comes to agile, the client/business team gets in the kitchen while you're cooking the meal and provides feedback while you cook. Furthermore, agile can result in a redistribution or change in project authority and decision making. These possibilities make waterfall advocates very uncomfortable. As a result, people are naturally at odds with agile because they don’t understand it but they know it means change and change scares many people.

Cameron: OK. I think that's a really good analogy there with the kitchen, as well. Now to move on a little bit here, you've worked more than fifteen years in tech management, leading both agile and waterfall teams. Since agile is roughly ten years old in terms of its surge in popularity, this kind of puts you in a prime position to elaborate on how the perceptions of agile and waterfall have changed during this time. Based on your experiences, how have the perceptions of both waterfall and agile changed?

Calvin: That's a good question. I was actually speaking with a colleague recently who provides waterfall certification training. As I chatted with him about expanding his training to include agile; he made it very clear that he was not interested. As if to say, "Hey, I don't trust it. I think it's the new kid on the block and I'm hoping it'll disappear sometime soon." His reaction is similar to what I see in some business executives. Typically, people love it or they hate it. However, what has changed is the adoption of agile and the reduction of waterfall. A rough estimate would be sixty percent of Fortune 100 companies are considering agile in some way compared to twenty percent five years ago.

Agile is gaining endorsements from the same global organizations that endorse waterfall. The Project Management Institute (PMI) recently started offering an agile certification (PMI Agile Certified Practitioner (PMI-ACP)) along with its highly coveted waterfall certificate, The Project Management Professional (PMP). The Capability Maturity Model Integration (CMMI) Institute also recognizes agile. However, the negative perceptions of agile still remain. 

Agile’s growth in popularity is promising; however, many still believe that it is a wild and crazy trend that will go away. Agile continues to get stronger because it’s awesome, and has helped companies that struggle with missed project deadlines and budgets. Many companies have started using agile because they are in need of something different. As a result, some remain skeptical because it redistributes a stakeholder’s power, changes the way individuals complete work, and has less trusted financial tracking mechanisms.

Agile will continue to grow because many companies and organizations are still struggling to do software effectively, efficiently, and meet customer demand. The skeptics will remain until agile proves itself. 

Cameron: OK, and you kind of brought up a really interesting point there. You talked about how that human perception, the human primal quality that there's kind of always the fear of the unknown—people are scared of change and people are hesitant to change. With agile, is it that people are scared of change, or is it they're scared it won't work?

Calvin: Well, it's a little bit of both. The reason why I started writing these articles, was to help organizations understand agile and waterfall. Some executives who introduce agile to their organizations are skeptical, they question if it's going to work. They want a method that is iterative and incremental, but they don't know if it's agile. 

With my Agile vs. Waterfall articles, I help these execs approach the problem in a systematic way. I want them take a deep-dive into the strengths and the weaknesses of agile and waterfall. Agile is not a panacea—one solution to fix all the problems. However, there are some things that agile does well, such as increasing team communication and transparency. However many are still struggling with "Hey, how do I measure the financial cost of it? How do I know if it’s working?” 

Cameron: OK, now moving on here again, you specialize in e-commerce websites, servers, software enhancements, and application development solutions. Does this industry have any unique properties that lends itself to adopting agile over waterfall methodologies, or vice versa?

Calvin: Yes, it does. The e-commerce industry has grown by leaps and bounds and customer demand is increasing with every new smart phone purchase. E-commerce is growing fast and absorbing the emerging technologies of tomorrow. We can see this from delivery drones, big data, smart watches, predictive analytics, health-trackers, the internet of things, and all the way to predictive shopping behavior. Customers are evolving and demanding shopping solutions with the next big thing integrated. Many retail companies are trying to keep up with this customer demands, not just for the sake of customer demand, but because market share is associated with this demand. 

Agile project management is a perfect fit for the e-commerce industry, because it enables organizations to keep up with increasing customer demands and aggressive opponents that threaten to take market share. 

Cameron: OK, and so kind of since one of the fundamentals of agile is being able to adapt, the fact that you have to adapt very quickly in e-commerce to stay above the latest and newest trends, it makes it a perfect marriage.

Calvin: Yes, it does. Many E-commerce companies are demanding agile. They believe it will help them better respond to increasing customer demands. However, many of these same companies are having a hard time integrating agile into their organizations.

For example, there are many traditional brick and mortar companies that are scaling their technology teams to keep up with e-commerce growth. They realize that they must evolve to a multichannel shopping experience. They know it’s important to be online and available when the customers want to shop. The growth is demanding new skills, methods and software management philosophies. Customers are demanding, "Hey, I want to be able to order online and pick up my product in the store that is closest to me." Well, that requires a lot of e-commerce work. First, you've got to integrate nationwide inventory systems with local inventory management systems and enable stores to ship from store-to-store in a cost effective manner. Second, you have to figure out how to use these stores as local distribution units similar to FedEx Kinkos. However, many traditional brick and mortar companies are just not comfortable using to this business model.

Cameron: OK. Now as one final question here, I'm going to ask you to put your Nostradamus hat on for a second. Waterfall enjoyed roughly four decades of project management monopolization before agile came along and kind of knocked it off as new kid on the block. Do you think waterfall will make a resurgence, or do you think it's more likely that we'll see an entirely new method emerge?

Calvin: Great question. I predict both will evolve, I'm already seeing hybrid methods take shape. In the meantime, I will continue to write and help managers with the current growing pains. I definitely think something new is coming. 

Cameron: OK, fantastic, and you've seen the new trends, if you can definitely get ahead of that and kind of brand it yourself, maybe you can get a chunk of that. Call it some sort of Calvin related name.

Calvin: Calvin methodology would definitely be ideal, so let's just see how it goes. Something has to give because companies are finding themselves with a new type of demand and an old way of doing business. Times have changed, companies need to respond quickly and do so in an almost predictive manner. 

Cameron: Fantastic. We look forward to the future iterations of your articles. Once again, this was Calvin Baldwin who spoke to us today about agile versus waterfall, and we'll make sure to definitely keep in contact about your future articles on LinkedIn. Thank you so much, Calvin.

Calvin: All right. Thank you.

Click here to read the first article in Calvin's series on agile versus waterfall.

Click here to read the second article in Calvin's series on agile versus waterfall, which deals with selecting a project method.

Calvin Baldwin has spent more than fifteen years in technology management. He has led agile and waterfall teams as they deployed hundreds of e-commerce websites, servers, software enhancements and application development solutions. As a consultant, he joins companies in need of software delivery change management as well as organizations in the process of increasing technical team productivity, business output and customer satisfaction. Visit www.CalvinBaldwin.com to learn more about Calvin and his work.