While Agile methodologies have made tremendous advances in programming circles, build and release management teams have traditionally shied away from a “full agile” approach. In informal polling of Build Engineers, I found only one team outside of my own that ran an Agile shop; while there are many out there who do, there are tender points that need special consideration.
A Build and Release Engineering team (which we'll refer to as Engineering Services) is an excellent candidate for the Agile methodologies. In the first place, our customer is extremely close to the development process - usually right next door if not in the same building. Much like application developers, time is the biggest constraint for Engineering Services teams. In our case bugs, administration of the build system, even instability (in less mature build systems and in-progress products), and the fluid nature of the products all contribute to this constraint.
Agile practices, and scrum in particular is the perfect fit for Engineering Services, assuming we acknowledge the same goals. Firstly, Engineering Services teams are customer focussed, service based, infrastructure development teams; our customers come first, the services we provide those customers must always be our initial focus. Secondly, we want to eliminate root-causes rather than create band-aid solutions. Lastly, we want to automate the repeatable and ensure the reproducible. Meeting these goals, and transitioning to scrum involves a proactive approach.
Our Method
An Engineering Services team is no different than an application development team. The scrum approach, process, and tools used are all the same. The real different is the amount of time that can be dedicated to features, as opposed to maintenance work. To achieve this, my Engineering Services team follows the process outlined below.
- Pre-sprint customer meetings
- Sprint planning and retrospective
- Daily Scrum + Commitment Review
- Scrum of scrums
- Sprint demo
- Backlog Grooming
Pre-sprint Planning
As part of the backlog estimation process, I pro-actively meet with each customer individually, prior to our sprint planning. Much like with application development, this meeting is used as a sounding point with product owners, a period where we discuss stories and understand the nuances, spelling out more detail if necessary. As of this writing, the method is scaling well with five separate meetings.
All our story tracking takes place in VersionOne, which allows our customers (usually development managers and product management) to manage and track their own ordered backlog. Customers add and remove backlog stories as necessary, reordering them along the way. During Pre-sprint customer meetings, we discuss the backlog items, in order to better understand the stories. I add product specific development (not features) stories into the Request bin, again enabling my own tracking, and group discussion with our customers during this pre-planning phase. The key point here is that each customer fully owns their own backlog, providing them the ability to control their own needs and priorities.
Once these customer meetings are completed, I organize the stories by overall priority, by estimating the balance needed to meet customer needs. When conflicts are apparent (usually due to compressed schedules or conflicting needs), I redress priorities with these varying product owners, ensuring an agreement is reached. As part of this planning phase, Engineering Services reserves a percentage of their time per sprint for infrastructure needs; this comprises of our own, internal backlog that helps us expand the system to meet our customer needs.
Sprint Planning and Retrospective
Engineering Services teams engage in the same sprint planning mechanics as application development teams. Our planning meetings start with a retrospective of the current sprint: what should we start doing, stop doing, and continue doing. We review the comments from the previous sprint's retrospective and determine how we improved, and how we can improve.
The next step is sprint planning, looking at the ordered backlog and determining what we can commit to, as a team. Stories are discussed, hour estimates are assigned, and team members pick off stories as they can. Team members are always asked “are you sure you can commit”, and the team as a whole is responsible for the sprint.
In our case, incomplete stories don't result in an aborted sprint, but a failed sprint. These are usually the result of last minute feature needs, or a decreased availability due to bugs, but underestimation or over-promising also contributes to these events. Rather than dogmatically “abort” the sprint due to a last-minute feature, we've chosen to integrate the XP concept of story trade-off, allowing our customers to eliminate stories to introduce others (where it makes sense).
Availability is the biggest problem facing Engineering Services teams, especially in the Agile ecosystem. Builds break, installers have bugs, infrastructure melts down... which all contribute to a reduced availability. The solution to this is two-fold. Firstly, my team plans different member availabilities based on product release schedules; for example, if two products are releasing in tandem and in the sprint being planned, the Release Engineer (also known as installer developer) will have little time. Secondly, we alter our plan to address infrastructure; ensuring a minimum percentage of our time is spent addressing infrastructure needs. This allows us to to address the root causes for build and infrastructure failure, in turn ensuring we have a higher availability for customer needs in the upcoming sprints.
Daily Scrum and Commitment Review
Like all good Agile teams, Engineering Services teams need to scrum, and scrum daily. The key, much like with other teams is adhering to the rules: talk about what you did yesterday, what you're going to do today, and what got in your way. Make it a point to Parking Lot non-reporting conversation and discussion, to ensure you run through the scrum; this helps the team quickly self-divide for conversation, and keep the process lightweight and engaged. Invite others to the scrum, if they add value. For the better part of three months, I had an application developer, and a team-lead from two separate teams attending our scrums.
Daily commitment is the other major takeaway that drives my team. Each of us commits to finishing something daily, in front of our team. We reinforce that commitment by writing it on the team whiteboard, exposing it to the public, and using it to drive our day. The next day we review the commitments. Did we meet them all? Why wasn't a commitment met? What could we have done to meet the commitment? Not only does forcing a commitment review encourage ownership and accountability, it also gives team members with the feeling of a job well done; it helps bring the idea of job completion to software engineering.
It's the nature of an Engineering Services team that emergencies come up. Machines break, builds require an unforeseen investment, or new installer features are injected at the last minute; all are possible. Anything that changes our sprint, anything that needed to be added into the sprint is done so in the form of a backlog story, added in VersionOne. This allows me to generate a Scope Creep report, in order to discuss what changed, and understand where our interruptions lie. These scope creep features have also been used as the inspiration behind backlog stories, or continue investment. They also help the team drive automation ideas, improving the Engineering Services product offering, while at the same time meeting the immediate needs of our internal customers.
Scrum of Scrums
A good Engineering Services team can affect multiple products, and is trusted to do so. We have several customers, several teams, and of course several users with which we need to co-ordinate our changes, and change plan. Again, much like with application development, the scrum of scrums provides the forum to do that. A representative discusses what happened last week, what's coming up this week, and most importantly what we're going to put in the way of our customers. While Engineering Services occasionally needs to affect product stability, or build expectations, the ripple effect can be felt across multiple teams.
Although the team most closely affected (i.e. the product developers and testers) might have “signed off, other teams may not be able to support a disruption of the planned sort. One great example of this is a shared component team, or product consumer. Perhaps, Project Management may have a time-line planned that doesn't allow for the planned disruption. The scrum of scrums provides a forum where cross-product (and if you're really lucky cross functional) teams can sit down and have these conversations.
Closing the Loop: Demo!
Our Engineering Services holds a demo of our work, at the end of each three-week sprint. We invite all of our customers to the same room (whether in person, or virtually) and work our way through all the stories, displaying and discussing the changes implemented during this time. This activity closes the loop on a story, and at the same time, provides an opportunity for the various stakeholders to learn about each others needs, based on our workload.
Unfortunately, many Engineering Services stories are difficult to show. Installer changes, code changes to the build system, these are all easy to demo; but infrastructure changes such as network rewiring are obviously difficult to show. For those stories, an Engineering Services team needs to mention what was done; there is nothing demo-able, unless you're lucky enough to impact build times!
One of the most common surprising outcomes of our sprint demo, was backlog stories. Having done some work for one team, another realized they could benefit from a similar investment, or had the same need. This approach has helped our team build credibility with multiple (geographically diverse) teams, as well drive future work and investment. While the demo helped close the loop story-wise, it also increased the amount of need, which is always a good thing.
Backlog Grooming
The end of one sprint demo blends almost seamlessly into the start of the next sprint. We hold our demonstration the afternoon prior to our sprint planning and retrospective, which means that after the demo, I (as the Engineering Services product owner) groom the backlog. Usually I'm weeding out stories that we happened to complete, removing stories that no longer make sense, or most commonly ordering stories.
I'll also add stories, based on either the outcome of the sprint demo, or perceived need. Again, this is where VersionOne helps. We use VersionOne to manage our entire backlog. We use the Discussion tab frequently, to ensure that we can track emails, or conversations associated with a story. I use VersionOne to order the backlog prior to the sprint planning session, which ensures that all our data is available publicly, allowing anyone at any time to see what Engineering Services is up to.
After the sprint demo, I order the combined backlog, in preparation for the next day's sprint planning. Negotiations are help with customers (if necessary), with the goal of entering the sprint planning session with an understanding of the absolute minimum customers need. As a rule, we always aim to surpass that minimum,
Wrapping it Up
While managing an Engineering Services team has its own challenges and constraints, introducing scrum is relatively straightforward. Issues will continue to develop, last minute features and bug-fixes will always be needed; but in the end, the sprint construct has freed up my team, enabling them to deliver higher quality products and infrastructure. The biggest key was training and coaching, it's what helped us get to the point we're at today.
About the Author
Chayim Kirshen is the Manager of Engineering Services at PlateSpin, a Division of Novell Inc.. As an experienced professional with over a decade of experience, his prior engagements have included the Telecommunications, Financial, Healthcare, and Education sectors. Chayim manages a combined build and release engineering team, planning and developing with a SCRUM evelopment process. He loves GNU Make, NAnt, and breathes python and revision control systems. Chayim can be contacted through his website, http://www.gnupower.net, or [email protected] .
