A Pony in the Pile - A Curmudgeon's View of SOA Adoption

Here's an old joke. A young boy (or girl) is bouncing with desire to have a pony of his very own. A frustrated father takes his son (or daughter) down to the stables to show their ignorantly enthusiastic offspring the realities of pony-ownership. Behind the stables, the father shows the young scion of his house a huge pile of pony manure. And, before the father can even speak, the child runs headlong and burrows into the pile. Appalled, the father shouts, "What are you DOING?" The muffled reply comes, "I just KNOW that there is a pony in here somewhere!"

I was reminded of this joke recently during a seminar that I prepared and delivered for a client. In just one day, I had to introduce a room full of system engineers to the essential concepts of SOA. In just one day, I had to say something meaningful to them that would help {sidebar id=1} them to plot a course for their systems, products and relationships with their customers. In just one day, I had to pour an ocean into a teacup.

The grinding of my teeth made my head hurt.

Standards, Standards Everywhere, But Which Ones Do I Really Need?

If you doubt that SOA is a complex topic, start by looking at the wide, wild world of Web Services standards (see Figure 1). In seven years, these standards have evolved into two distinct generations, the core standards and the WS-* standards. Each generation has three levels: enabling standards (e.g., SOAP & WS-Coordination), enhancing standards (e.g., WS-Addressing & SAML) and utility standards (e.g., SOAP-RPC, WS-AtomicTransaction, and BPEL4WS). Many of these standards have reached, as of mid 2006, their third revision birthday - an important milestone for standards maturity - meaning that you have to be careful to check the publication date of everything that you read. There have been (and still are) competing standards. And, vendor APIs (which are not governed by the standards) are all over the map.

                                                       Figure 1 - Map of Web Services Standards

                                                       Source: Valtech

It is enough to daunt even the most technophilic soul. Do you really have to understand all of these standards? Do you have to adopt them all at once? Some of these standards are intended to be adapted and extended by the system implementer. Which ones do you have to build? Which ones can you buy? The vendors have been strongly driving the standards activity. How far can their products be trusted? Do they really have your best interests at heart?

Few, if any, organizations can absorb all of this technology change at once.  Considering that the Web Services-for-SOA marketplace (as distinguished from the Web Services-for-RPC marketplace) is only now crossing the chasm to the early adopter phase, no responsible IT leader would try. An effective SOA adoption strategy must start with a pragmatic and evolutionary attitude to SOA technology introduction. It must address the questions: What do we need now? What will we need next? What can we leave to later?

It is enough to daunt even the most technophilic soul. Do you really have to understand all of these standards? Do you have to adopt them all at once? Some of these standards are intended to be adapted and extended by the system implementer. Which ones do you have to build? Which ones can you buy? The vendors have been strongly driving the standards activity. How far can their products be trusted? Do they really have your best interests at heart?

Few, if any, organizations can absorb all of this technology change at once.  Considering that the Web Services-for-SOA marketplace (as distinguished from the Web Services-for-RPC marketplace) is only now crossing the chasm to the early adopter phase, no responsible IT leader would try. An effective SOA adoption strategy must start with a pragmatic and evolutionary attitude to SOA technology introduction. It must address the questions: What do we need now? What will we need next? What can we leave to later?

Here is where our agility in SOA adoption begins.
 

Planning, Schmanning!

But, you've gotta have a plan! ... Don't you?

Ken Schwaber, the founder of the Scrum process, tells a story about describing traditional software development and project management practice to the process scientists at Dupont ... and of being laughed at. [1] You are making, they said, a fundamental mistake of confusing an empirical process with a planned process. A planned process is one that you have executed over-and-over, thousands of times, like building tract homes. You have performed the process so often with so little variation that you have a firm statistical basis for planning and optimizing the process and, consequently, a high assurance of meeting your objectives. The software development process is not, they said, such a planned process.

No. The software development process is, more likely and more often, an empirical process. A process which, though it may be similar to projects that you have done before, has enough novel features that it is more like oil-and-gas exploration than home construction. In exploring and exploiting an oil-and-gas field, you make your plans and then you adapt them to the real conditions on the ground. You quickly learn: most fields are similar; none are the same.

Let there be no doubt: SOA is not like anything your organization has done before. SOA is an enterprise architecture philosophy. SOA is a system architecture philosophy. SOA is a new analysis discipline. SOA is a new design discipline. SOA is a new testing and quality discipline. SOA is new technology, new tools, and new infrastructure. SOA is business process change. SOA is a culture change. 

SOA is not just Web Services.

In technology and organizational change management, many initiatives are either under-planned or over-planned. They are planned in ignorance. They are planned without clear goals and reasoned objectives. But, more to the point, they are often statically planned rather than dynamically planned.

One of the distinguishing characteristics of agile project management is how it embraces change, how it lives and breathes adaptivity and feedback. Agility is planning and execution; it is not just a plan and a whole lot of work. It is a process of goal-setting, goal-resetting, prioritization, reprioritization and always measuring your progress-to-goal and assessing the real business value delivered. It is not just the consumption of budget and schedule.

Long Jump Over The Grand Canyon

It is exactly this agile attitude toward, and discipline of, planning that we need in order to make SOA adoption work. The plan is worthless; the planning is priceless.

The Project Management Institute defines[2] a project as an endeavor with a specific beginning and ending in time that works towards an achievable set of objectives. A program is different. A program is a coordinated set of projects that collectively advance the organization toward a set of business goals. A program may have a beginning in time, but often doesn't have a defined end. A program has goals and strategies, more than plans. A program doesn't know its exact path to its goals, nor if it can achieve them wholly. It must explore the ground and manage the risks as it encounters them.

I see a lot of organizations in their track shorts and tees limbering up for SOA adoption. They pump their knees high, shake out their arms and hands, huff a time or two, take a deep breath and sprint toward the sand pit to make a long jump into the IT record-books. Unfortunately, they often haven't taken a good look at the sand pit. The SOA sand pit is a canyon - a big canyon. Onward at breakneck speed, they run. With a mighty grunt, they leap. With dawning realization, they holler as they fall out-of-sight, "Oh!  OOOOH! CRAAaaaaaaa <splat>!" Another greasy spot on the rocks below.

If you survey the canyon with binoculars and an experienced guide, you will find out that there are several paths down into the canyon and back up the other side. These paths bring you up on the north rim at different points. But, they all have this in common: first you work your way down, then you cross the river, then you work your way up.

There are three stages of maturity in SOA. 

Adoption starts with service-oriented (re)integration (SOI) of existing systems. At this stage, we learn to identify and package Web Services. We learn the difference between data-oriented, function-oriented and process-oriented services. We build services to wrap legacy systems in order to protect our IT investment and to standardize and to stabilize interfaces so that we can build future systems on a firmer foundation. We identify systems-of-record and build data integrity mechanisms to synchronize automatically the three or four extant (and inconsistent) copies of our data. We explore how to deploy Web Services into real networks, learning that service port/endpoints and service implementations are not always in the same place in the network and learning that UDDI is too bloated and WS-Inspection is too stripped down for real use.

The next step is the enterprise service bus (ESB). At this stage, we realize that point-to-point connections in a service-oriented architecture lead to an unmanageable, unsecurable rat's nest of services and consumers. We realize that asynchronous, reliable messaging has some major attractions - if it can be configured into an efficient, distributed network. We come to understand that having a single security enforcement point at the bus is worth its weight in gold if we are serious about security. And, we learn the real value of a service directory: static load-balancing, fail-over, release environment management and notification event management.

Ultimately, we climb upwards toward business process management (BPM). We come to understand that applications and services are but cogs in the machinery of business processes. Business processes are the mechanisms that realize business value. Eventually, we realize that the capitalization of IT projects, the management of IT project and system portfolios and the optimization of enterprises in the age of ultra-competitive markets fundamentally depend on making the business visible and flexible in a way it has never been before.

As we work our way down into the canyon, we don't know what dangers we will encounter. Crossing the river will take a great effort and much care. Climbing up the other side, those canyon walls do surely look endlessly high. We may have to try a couple of different routes before we find the one that is best for us. Our goals will change during each stage of the journey. So will our strategies, plans, and schedules. Count on it.

Is it worth all of the effort? Is the North rim really that much nicer than the South rim? Well. There isn't much water around here and the vegetation is a bit sparse. Through the binoculars, we think that we see green trees on the other side. And, at least, there is a river down there. Hmmm. Maybe we should give it a shot. And, even if we never make it all the way to the other side, we can still follow the river on downstream to new lands that we haven't even dreamed of.

Agile Is As Agile Does

All right! I never said that there wasn't a pony in the pile. I just think that it is undignified to jump into the pile head-first. It is a really big pile. There has got to be a better way of finding the pony. I believe that the extension of agile principles to program management is the best  and only way to do pile management and pony development for SOA.

Agile program management is:

• Explicitly goal-directed and sub goal-structured - acknowledging the stages of maturation that are needed to reach interim and ultimate goals.
• Business value-focused in its goals, its progress measures, and its deliverables - emphasizing that working systems are the primary measure-of-progress and deliverables.
• Feedback-driven and adaptive in its planning and execution processes - featuring regular and frequent assessment of progress-to-goal (i.e., at least quarterly for a three to five year program).
• Incremental implementation and refinement - where the essential philosophy is to implement the adequate, refactor to the good and evolve toward the perfect.
• Stakeholder-oriented and role-based - ensuring that all of the parties with vested interests in the status quo and in change are represented and engaged in the process.

Finally, if we do find a pony, will it be the perfect pony of our dreams? Not bloody likely! But, it is likely to be a real pony. A flesh and blood little workhorse that, though it produces a lot of manure, also hauls a lot of freight.

[1]       Schwaber, Ken & Beedle, Mike,  Agile Software Development with Scrum, Prentice-Hall, Inc., Upper Saddle River, NJ, 2002, pp. 24-25.

[2]       A Guide to the Project Management Body of Knowledge, Project Management Institute, Newtown Square, PA, 2000, section 1.2 "What is a Project?".
 

About the Author

Al Goerner has been on both sides of the Web Services/SOA world, as both a vendor and a consumer/provider. He has been Chief Architect at two companies, one that built a first-generation Web Services infrastructure product for business-to-business integration and at another which built message filtering and secure communication services. Today, Al is Director of Product Development and Intellectual Property in the Skill Development division at Valtech Technologies Inc. (http://www.valtech.com/).  He works with clients on agile methods & processes, object-oriented analysis and design and object- and service-oriented architecture.