How to Evaluate CM and ALM Tools

Evaluating and selecting configuration management (CM) and application lifecycle management (ALM) tools is an opportunity both to ensure that you have a good process in place and to learn the present state and the state-of-the art in CM and ALM technology. As with most technology, as you move from one generation to the next, the capabilities grow more rapidly, while the timeframe for each generation shrinks. Evaluation is a critical step in securing good process, and should be done in a manner that will ensure success. You don't want to start any project with the wrong requirements, nor with a dated understanding of technology.

First Things First
What's the best way to go about tool evaluation. Before you start evaluating, or at least as part of your initial preparation, make sure you equip yourself in the following three ways:

1) Know your users: talk to them. What would make their life easier. Your users don't know exactly what they need. But they might have some complaints, suggestions or other hints that will let you understand them. User complaints are normally high priority - and they should be. But some are objectively more urgent than others. Still, your users are your customers, so find out what you need to make them happy, and then plan to exceed their expectations.

2) Understand today's CM and ALM technology. It's not the same as ten years ago. It's not even the same as five years ago or three years ago. Yes, many tools stay the same, substantially, over long periods. But some are charging forward. And there are some new kids on the block too. Use the vendors to help you catch up. They say they've got some ground-breaking technology - make them tell you why in an hour or two. Read the CM Journal (especially the column CM: THE NEXT GENERATION).

3) Understand CM. There are many definitions of CM. But the scope of CM is so wide and varied, you need to understand where you stand. Are you interested in Software CM? Good. For traditional or agile processes? Look at both. Ask why you can't apply that agile process to your traditional environment. You may be the world's foremost expert on merge tools, but you might discover that one of the keys to successful CM is to minimize branching and merging, not to cater to the most complex scenarios. Maybe you're not interested in software at all? Maybe you want to understand CMII (icm.org) or ITIL. Maybe you're interested in asset management. These are related but different areas. You need to understand the global scope so that you don't, as I have done on a few occasions, go down a whole 15 minute conversation only to find out that you're talking about two different things.

What Next?
Now it’s time to really take a look at what's out there. A lot of CM managers will look at a handful of tools, and then choose two or three to investigate in more detail. While you may get lucky this way, you can easily miss the tools best suited for your environment. Evaluation is a time-intensive process. Don't make a mistake up front that may cost you more time down the road.

Cast a wide net and read a paragraph or two about every tool. Pick 15 to 25 tools. Spend a half day or a day browsing through the web site for each tool, say 15 minutes per tool. Spend enough time to identify things you like and perhaps things you don't like. By the end of the day, you should be ready to start with a list of about six to eight vendors that you want to contact for more information. This may be a formal step, as in an RFI, or an informal step, as in, "Can you arrange a demo of your tool over the web?" If you're doing a formal RFI, spend the time FIRST to have some 1 or 2 hours web demos presented to you. In one week's time, you should be able to take your initial set of tools and have a good idea of which tools are candidates. But even more importantly, you'll have a better idea about what features and criteria are most important. This in turn will allow your RFI to be more intelligent.

During this week of demos you're going to learn a lot about tools: How wide-ranging they are; how user interfaces can vary; how the scope of one tool is far more or less than another. You probably won't have a good feel yet for things like performance, administration and customization, but the basic CM/ALM functions, traceability, and ease-of-use should make themselves apparent during a demo. At the end of each demo, ask the following questions:

1) How much will it cost, per user, all in, for 25 developers and 25 occasional users? For all the ALM tools if possible.
2) Can I install the tool to evaluate it myself? Do I need any special infrastructure or training? How much?
3) How many support staff do you recommend for administration and customization per 50 users, as in (1)?

Both for CM Process and CM Tool Suites, there truly is a Next Generation. Next Generation processes and tools come primarily from those who have spent a long time with the previous generation, and have the technology at hand to improve on them. Vision is important. But capability must be there as well.

The scope may be bigger than you realize. But that doesn't necessarily mean the solution must be more complex. For example, if I need a way to create driving directions, there are a host of tools that can do that for me. Now let's increase the scope. Why do I want these directions?  To help myself and others to get to their destinations. OK, well, what if I could generate the instructions and make it easier to follow them: the GPS is born and it’s easier than a map or driving directions to use. What about avoiding traffic to make my drive more efficient? The next generation of GPS evolves.

If I just looked for the best way to generate driving directions, I would be dealing with issues such as: What's the best format to print them in.? What should I highlight? Should I have some back-up routes in case of construction or traffic jams? How should I file/store these, (assuming I'll need to use them again)? These are great questions, but the next generation GPS makes all of these totally irrelevant. It's the same with CM/ALM. If you don't start with advanced technology, your processes are going to deal with things that should be totally irrelevant.

What's next is to clearly understand what is the current state-of-the-art for CM/ALM, whether it's process technology or tool technology.

Hot Topics for Next Generation CM/ALM Tools
Here are a list of hot topics you need to be on top of. These deal with issues which are pertinent to those looking at evaluating CM/ALM tools. Many of them have been covered in the CM Journal.

CM for agile development: Even if you're not doing Agile Development, tools that support it will give you an edge in fine tuning your traditional development methods.

Agile CM: You don't want just your development to be agile, you want your CM to be Agile as well.  This should make it easy to adjust to your CM requirements, lean, though not less functional, and naturally integrated into your user activities, not an overhead. Agile development tends to be more prioritize/assign oriented than traditional development. Your CM/ALM tools should help you manage priority-driven activities.

Global (i.e., multiple site) development: CM is a long term application. Odds are that you'll be bought out, move, or buy out other companies in the next 10 years. In any of these cases, along with the traditional scenarios, multiple site development will be critical. Rather than complicating your CM infrastructure if done right, though, it will simplify it.

Simplifying branching and labeling strategies: Branching is overloaded. Older tools cater to managing the overloading. Newer tools have first order objects and algorithms to handle things such as promotion, parallel checkouts, change and/or feature packaging, build and baseline definitions, so that you don't have to develop complex branching strategies and perform tedious labeling. Better yet, some tools will even let you know when you need to branch or merge.

Seamless integration of CM/ALM functions: When you buy multiple tools, you establish a permanent glue team that glues them together, manages upgrades which break the build, and continue to grow the glue so that you have better integration. Tools which seamlessly integrate the CM and ALM functions, sit on a single repository, with easy traceability navigation and unified user interfaces. Training and administration costs are not multiplied by the number of functions.

Roles and dashboard reporting: Role-based operation is essential from process, to data requirements, to user interfaces. Tools which clearly recognize this are designed for the many roles, not for one-interface-fits all complexity. Furthermore modern tools have various dashboards which are role specific and eliminate much of the menu navigation and report generation held over from the '90s. The more advanced dashboard technologies allow drill-down from summaries and graphs, and also let you dynamically modify the key parameters of the dashboard so that you can inspect, for example, multiple releases of multiple products, or compare all the data for arbitrarily selected builds.

Requirements traceability, interactively: An integrated ALM suite will contain requirements, and also test cases. You are able to rapidly trace from a test case to a requirement and vice versa, as well as identify holes in your test case coverage. More advanced tools will tie in the concept of test runs and relate them to CM-tool tracked builds, so that you can ask questions such as: When did this test last pass/fail? What percentage of requirements have passed testing in a particular release stream? Better yet, all of this is done through a few interactive clicks with zoom-in to details.

Increasing user productivity in CM/ALM processes: It used to be the case that you had to convince your users to use the CM/ALM tools. If this is still the case, you need to change them out. Next generation CM/ALM tools provide advanced functions which simplify a user's world. Take the developer for example: workspace synchronization, history of work performed in the past year, single-click check-in, delta report and promotion operations on multi-file changes. Some tools will even let you click anywhere in the code tree to search the code in that tree, or perhaps search through all revisions of the code. Active workspace technology is used by some to warn users, visually, if they're missing files from the workspace or if their workspace files differ from their context view. A simple thing that can save a lot of grief. Manager and executive productivity improvements are even more dramatic.

Process-centric tools: You not only have to follow process, you have to prove that you do. Process-centric technology not only documents the process, but helps to enforce it, typically through permissions, rules and triggers on state transitions. They also make it easier to present data to each user in terms of a set of to-do lists, or inboxes (e.g., to be reviewed, changes in progress, awaiting approval, etc.). Process engines of modern tools span all of the CM/ALM functions so that the same process behavior is common to all functions, simplifying the learning curve.

Change-based configuration management: If your CM tools don't support change packages, or, almost equally bad, make you use a separate tool (from your version control tool) to define and manage changes, you're practically living in the dark ages. Developers don't create file revisions;  that's not natural. They make changes: fix a bug, add a feature, clean up the architecture, etc. If you have to check in 10 files (oops, I forgot one), or do 10 delta/difference operations, or promote 10 different files, or figure out how to merge the changes made to 10 files to a different release - you don't have to wonder why your developers don't like CM. On the other hand, if your developers can scan 20 change descriptions rather than 100 different file revisions, they'll find the CM tool helpful. If your CM managers are managing file revisions in their baselines instead of promoting or rolling back changes, you'd better hope they don't change jobs or go on vacation.

Workspace management: You have file in your directories. How out-of-date are they? How do they get resynchronized with the latest development? Are there new files missing from my workspace? Are any of these files checked out by someone else? Which files are R/O and which are R/W? How can I compare my workspace contents with the build that went out last week? The answers should be one click away, or better yet, visually apparent though active workspace functionality.

Rapid tool deployment, data loading, evaluation: Some tool vendors are great. They'll come in, do a demo, install their software (maybe even on their own machine that they'll lend you), load in some of your data, and help you through the evaluation, pointing out some key benefits of their solutions. Others will say: it's 5 minutes to download and install. We can do a live web-based demo at your convenience and repeat it for those who can't be present. Click the “load software" button to capture your existing data and the "load problems" button to capture your existing problem reports. Use the Quick Start guide to run through the evaluation on your own data, and at your own pace - our support team is ready to help at any time. I get concerned when the vendor has to come in to do the dirty work. I think: consulting costs, high administration, complexity. If I do it all quickly myself I say, "I get this!"

Reliability, availability and disaster recovery: Your CM/ALM repository is not a developer tool. It is a product team tool, and perhaps even a customer accessed tool. Down time is costly and does not make for a good impression. The inability to access data will preclude you from some potential customers. And what happens to the business should disaster strike, even if it's just a disk crash - sure you can recover from backups... just more down time. This should not be an issue today. CM/ALM apps are serving hundreds of users who need continuous access to the data. Disaster recovery needs to be on warm-standby, not days away. And remember, the weakest link in one ALM function can bring down the whole suite.

Low administration: How big did you say your CM admin team was? Times the average salary? That's expensive. What happens if half of them are off with the flu? Or if a couple of them get hired by a new company? With next generation administration, as demonstrated by a handful of tools today, CM administration should be a non-issue. If your tools require constant tuning, partitioning, recovery, or any other sort of administration, ask why. You should be able to manage a couple hundred users with less than a full time administrator. You should be able to train your administrators in less than a day. If not, you don't have a good administration user interface for your tool suite, or else you have too many different administration user interfaces for the different functions (version control, change control, problem tracking, requirements management, etc.).

Customization to corporate processes: I know of one CM/ALM vendor that has everything pre-packaged and working very nicely, just don't ask for changes please. Of course they're willing to provide them for a per diem fee.  How many diems did you say that would take? Next generation customization of CM tools has a long way to go in the industry at large, with perhaps a couple of exceptions. Don't look so much at how closely the tool fits your process. Your process will change. Even if it doesn't, it will be different in another part of the organization. Yes, good out-of-the-box process is an important criteria. Easy to customize tools are even better, especially if you can use the same customization tools across the entire suite (and I don't mean a C compiler).

Customization of process, data schema, guidance, and user interface. If you want continuous improvement to meet your evolving agile processes, this is a key consideration.

Build automation, release and deployment management: The assets are in the repository, now I need to get them out. I want to automate builds so that I can run them frequently. In fact, I want all team members to be able to run them easily. I want to be able to deploy, not just to web sites or customer areas, but to my own testing and production machines, not to mention individual developer workstations so that they can do adequate "unit" testing of their changes (unit = change package here). Managing builds, releases and deployments goes much further still. I want to know what's the difference between what's currently deployed and what I'm about to replace it with - problems fixed, new features, how many changes, functional audit results, etc.

Establishing contextual views of CM/ALM data: A local company I spoke with was doing 2500 builds nightly. Not a big problem - it was all automated. Now, one of the builds has a problem. How do I zero in on the build? How do I put myself in the context of the build so that I can rapidly ask - what's changed since the last successful build? What high-risk problems were addressed? What version of that file was used for deployment? These are all context-specific questions. The context may be a build, it may be the latest view of a product development stream, it may be a baseline or something more custom. How easy does the CM/ALM tool let me select my context? Does it help restrict my queries based on the context? Does it figure out where I need to branch from? Does it fill in the context sensitive portion of my forms and dialogues when I click an add button? Does it automatically capture information so that I don't have to remember to label that data?

Be Not Afraid
Evaluating is a daunting task. How many dozen tools are there? And each vendor says it's tool is the best! The first temptation is to stay away from anything that does more than I think I need. Resist that temptation. Especially with your first couple of evaluations. Instead, look at a couple of full ALM solutions first. Get the big picture. Then it's much easier to make the decision. If you're afraid of the complexity, you won't know two things: Am I covering the requirements adequately? What complexities am I introducing by leaving out part of the solution?

If there are areas that you don't understand, bring a couple of others in from the team, especially those who are familiar with the broader application lifecycle. Just get them to sit with you through a couple of 90 minute presentations and ask for their feedback. Understand the big picture. And vendors are a means of gaining that understanding. Ask them the hard questions, or if you're not comfortable doing so, put them down in your RFI. Let others participate in forming the RFI so that you have an adequate range of questions.

Your RFI is not going to get you answers to all of your questions. What it will do is help you identify the vendors that are worth short-listing. Most vendors will say yes to virtually every one of your requirements. But a yes for one will mean "Click on Preferences and select..." while for another it will mean "We'll send two guys in and by the end of the week (and maybe the end of your budget) we'll have that set just right..." Focus on the quantitative questions: How many administrators should I have? When will I hit a scalability focal point when I'll have to upgrade my hardware, etc.? How long would it take to create a trigger to do ABC, or to add a menu button that does XYZ? Get a feel for the complexity and level of administration. Ask for reference sites and hit them with the same sort of questions.

Even better is to ask if you can install, load up, and evaluate the tool yourself. Do you need training ahead of time? How much? Can I try it first without training?

If you've not evaluated CM tools before, you're about to add to your expertise. If you have already done so before, and are looking mostly for new players and new releases, you know how much expertise you will gain through this process. It is always a technology learning experience. Hopefully it will be more: a decision-making experience based on sound research and feedback from your team.

Money Talks
How confident is your supplier that theirs is the best CM/ALM solution? They have invested part of their career into selling the product, but that could have been the only available job. What about if we put 10% down and pay the rest in two more installments over the year or walk away from it if we don't like it? The supplier wants your business. If they believe in the solution, this shouldn't be a problem. If it is, raise a red flag and suggest that you'll move on to the next solution.

Even if their solution is not the best, many vendors will let you jump through the hoops, using up your evaluation resources so that you don't have time to get to the better solutions. So you really want to focus first on those tools that won't eat up your resources:

• There's no pre-requisite infrastructure to buy and install
• There's no up- front tool cost
• There's not a large investment of your time in the evaluation
• There's minimal training and customization costs for the evaluation

Be reasonable. If someone wants a few day's wages to come in and understand your process and customize the tool to that process prior to your detailed evaluation, that's good money spent. If someone wants to put one or two people on your staff for the course of the evaluation, that's fine, as long as it’s at their cost and not yours. If someone wants to run the evaluation for you, that’s a red flag. They may be trying to help you or they may be trying to hide complexity from you. Identify which before proceeding.

If the contract is non-trivial, a vendor should be willing to invest a few hours to a few days pro bono. Rather than taking a customization course, have them walk you through a customization effort that they can perform for you.

The bottom line is that as long as you hold on to your money, they have to sell you the solution. Withsuch a backbone technology, you really shouldn't let go of the money until your team is up and using the tool. Start with a pilot if necessary, but make sure you know what you're paying for and what the total cost is. I've seen too many companies spend all of their money on the best solution, only to find that it was a money sink which mostly meets today's needs, but won't grow into tomorrow's.

Wrapping Up
Evaluation is a bit more complex than you may have thought. Put the effort up front, so that you understand what you need and don't commit your resources until you're sure it is a working solution in your environment. If you hit a tool that's expensive and/or time-consuming to evaluate, set it aside and go on to an easier one, so that you don't chew up your resources on the difficult ones.  These are probably hard to use and costly to administer anyway and you can always come back to them if you don't find something better.

Finally, if you're looking for more detailed evaluation criteria, sorted by CM generations, checkout the articles in the November 2008 CM Journal.