In a medium to large software shop, where several groups are involved in the development of a product, implementing an agile methodology may be a challenge.
One of the most challenging situations involving adopting agile is when doing so in a software shop that has several specialized groups already in place forming silos: development, quality assurance (QA), business analysts (BA), software configuration management (SCM), documentation, architecture, database admin (DBA), and user experience (UX). These shops may or may not have fixed (non-negotiable) delivery dates with a very tight schedule, developing either commercial products or turnkey solutions for customers. The goal is to take these silos and form a cohesive team while delivering useful software by the required date.
By using Scrum as the baseline methodology, the goal can be achieved in a way that will help the transition from the old, silo-oriented organization to the new, team-oriented organization.
The Actors
It is critical to map the three most crucial Scrum roles—product owner, ScrumMaster, and the team—to the traditional organization in a way that doesn’t feel too threatening to the current managers.
Product Owner
Given the responsibilities of the product owner, it is quite natural for a member of the business analysts group to take on this role, as they are the ones already generating the requirements. For complex projects where multiple BA members are needed to gather the requirements, one BA member must be selected as the “voice” of the group and act as the product owner for the team. This BA member will be responsible for maintaining a ranked backlog for the team to work on.
It can be tempting to assign this role to the BA manager or supervisor. This would be a mistake, as product owners must focus entirely on the product without the distractions of the group dynamics, such as tracking and reporting, sick leaves, vacation, conflict resolution, etc.
ScrumMaster
It has been common for project managers to be repurposed as ScrumMasters. This is a good idea as long as they have received the proper training and embraced agile. You get expertise on the inner workings of the company and knowledge of the team members for free.
Regardless, it is better at the beginning of an agile adoption to have at least one hired coach who helps these new ScrumMasters ensure that things go “smoothly” during the transition.
How many ScrumMasters do you need? It depends on the strengths of your project managers. Some project managers can serve as ScrumMaster for more than one team, while some can only work with one team. The organization must make sure that the coach is not overloaded; coaching is hard, and if you need to coach too many people at the same time, you lose your effectiveness.
The Team
The team must include all the groups that are directly involved in the day-to-day activities crucial for the software to be delivered. These groups include developers, architects, QA analysts, BAs, and documenters.
Unless the organization is pretty small, it is not realistic to have all members of each group in a Scrum team as the team may become too big to be effective (as a rule, the team should not be bigger tan nine members). In that sense, it is better to have several Scrum teams coordinated using scrum of scrums, where each Scrum team is composed of members from all the functional teams.
Ideally, each Scrum team is composed of four developers, one QA engineer, and one documenter. This gives a total of six people on each team, plus the ScrumMaster. This mix can tackle about eight midsized stories (3-4 days each) per sprint, so each team can provide enough visible value by themselves. Per the Scrum rules, team members must have time dedicated exclusively to the stories accepted by the team. This composition leaves some slots open in case the estimated workload for QA or the documenter requires the addition of another team member.
If there are more QA engineers and documenters than Scrum teams, it may be useful for them participate in the daily scrums as observers. This allows them to easily enter a Scrum team as needed (e.g., to replace someone on vacation) and helps prevent the feeling that it is a “them and us” (those who are in a Scrum team versus those that are not) scenario.
There are many activities that QA engineers and documenters could be doing that helps the common purpose, including: Write Manuals that are not directly related to the stories being developed and working on testing automation, load testing, and performance measurements, etc. These activities could be controlled using Scrum and their progress reported in the scrum of scrum. That way, everybody feels that they are in the same boat.
The Acts
Having established the roles, we need to know how each group fits into all the activities and meetings that Scrum defines.
Notice that the architecture, SCM, UX, and DBA teams have no place in the roles of Scrum. This is because they play more of a support role than an operative role. This does not mean that they are not needed; if the organization has these positions, it is most likely that they are needed. Their contribution to the process will be outlined in the following sections.
Backlog Grooming
Per Scrum rules, backlog grooming is the sole responsibility of the product owner. For large software products, there will be many BAs gathering requirements and writing stories, but only one keeping the ranking of the backlog.
After the requirement gathering and during the story writing, the UX and architecture team become involved. When usability is a must, the UX team can build some prototypes to show to the BA. If a public API is maintained, the architecture team can do the high-level design of the API to jumpstart the discussion with the team. The architecture team and DBA can start thinking about the possible impacts in the data model and entities of the system. All these activities are preparations to jumpstart the discussion with the team during the sprint planning.
There is one last missing detail: Who does the story estimation? If you have only one or two teams, go ahead and put them all in the same room and spend a whole day estimating stories. This can be done either during a sprint (adjust the velocity accordingly) or just after the sprint (this works best if the sprint is two weeks long).
If you have more than two teams working on the same piece of software, then you need a different approach. The good thing about big systems is that the same kind of activities keep appearing time after time. By keeping track of these activities and the history of estimate versus real effort, you can build a simple estimation model that allows the “quick” estimate of the activities, or you can try an estimation model based on impact and complexity of the change. The responsibility to build and maintain this model can be given to the architecture team. Whatever the approach, the model must be revised by the team from time to time in order to guarantee that the results are acceptable.
Sprint Planning
During the sprint planning, the BA, architects, and UXs present the team with the stories and their initial analysis or design. The team then breaks the stories down into tasks.
If for some reason the team feels that the original estimation is too far off, then the estimate must be adjusted accordingly. If this happens too often, or with too many stories, then it must be tackled in the retrospective. Perhaps the model must be revised, or perhaps the architects either didn’t understand what was needed or failed to properly explain what needed to be done. Of course, this changes the scope of the sprint and triggers a revision of the whole backlog estimation.
Retrospective
All the lessons learned during a retrospective must be passed along to the PMO to update the process standards accordingly.
If possible, a PMO representative should be present as an observer or, better yet, trained as a retrospective facilitator. This gives the PMO a bigger involvement in the agile initiative, and provides them with “from the trenches” information.
Life Outside Scrum
There are many activities that are not part of the Scrum process that are needed to fulfill the bigger purpose:
- The architecture and UX teams have the responsibility to define an initial set of standards and evolve them with the team as the projects evolve. This becomes critical when the product increases in size and scope needing more Scrum teams to be involved, as standards help the inter-team communication by providing a common language to speak, specially when working on commercial products (which tend to outlive the developers).
- The BAs must gather requirements from both the internal and external clients to feed the backlog.
- The SCM team must keep tuning the build scripts and the continuous integration server using the team’s feedback.
- The QA team must run tests outside of those that validate the stories being developed (i.e., performance, stress, and regression tests).
- The DBA team must work with QA and architecture to identify problems in the database performance and propose solutions to be included in the backlog.
- In an organization that sells software to third parties, the sales team must work with the BA to add to the backlog any requirements it gets from prospective clients and from watching the competition.
Continuous Improvement
The journey has just started once you have put all the pieces into place. Each retrospective will give useful insights into what is working and what is not. Expect failures, especially during the first sprints. Learn from them, adapt the process, and move forward.
Just keep in mind what agile is about: teamwork, producing valuable software, being able to embrace change, and trust.
