Overcoming the Fractured Development Cloud

The shift toward agile and cloud methods results in a variety of options that empower developers to choose their specific development processes, platforms, and tools to meet their design objectives. That’s great for project teams, as it enables them to make rapid decisions around team structure, tooling, processes, and technologies all of which contribute to improving delivery speeds. The downside is that there is a proliferation of incompatible tools and processes that sprawl across the modern enterprise as organizations have gone agile, cloud, and global. In addition, development environments are increasingly being optimized around project goals and objectives, with less focus on the improvement of program-wide and cross-organizational application lifecycle management. This results in incompatible databases and data formats, inconsistent processes and tool configurations, and high support costs in the enterprise; achieving a coordinated enterprise delivery and deployment system is difficult at best. I refer to this phenomenon as the “fractured cloud.”

There’s a reason this “fractured cloud” is proliferating in the enterprise. Something is missing in our conversations—that critical piece: “the collective intelligence of the cloud.” That is, “the community architecture” that allows entire organizations to come together and gain enterprise-wide leverage needs to be addressed.

The Collective Intelligence of the Development Cloud
When Brian Behlendorf and I founded CollabNet in 1999, the question we sought to answer was “What was the magic in the burgeoning open source industry that had enabled a very small set of Apache software developers, most of whom never met each other in person, to create a web server that became number one in the market at literally fractions of the cost of commercial alternatives?” Fundamentally, we believed that this open source magic had less to do with the fact that open source software is free. Rather, we believed that the ability to create market pervasive and high quality robust software at a fraction of the time and cost had everything to do with the following guiding principles:  

• Transparency: The unrestricted transparency and ability to easily access, collaborate on, and extend a software application “in the cloud” from anywhere on the planet
• Collective Intelligence: The passion and leverage gained by enabling any developer to discover and extend another developer’s applications, resulting in capabilities far greater than what they could achieve by working in isolation
• Process Codification: A platform architecture into which collaborative software development processes, such as continuous integration and delivery, can be codified into repeatable and shareable tool templates that can be instantly provisioned and shared across a global enterprise
• Cloud: A web-based platform that breaks down the barriers of silo’d and loosely integrated tools and development processes that is optimized for distributed and collocated software development teams
• Openness: A flexible and extensible development cloud that drives a culture of collaboration across enterprises utilizing a heterogeneous mix of tools, processes, and technologies, while addressing an array of industry and design team objectives
• Community Architecture: A hierarchical development “forge” into which companies can map their projects and IT assets into business lines and technical architectures using flexible project hierarchies

Extending Cloud Development to the Enterprise
We sought to extend this open source magic by providing a platform which incorporated the above guiding principles to commercial enterprises. We wanted to provide developers and CIOs alike with the development productivity, cost reduction, and governance enabled by the “cloud,” although we didn’t know what to call it back then. There weren’t any analyst magic quadrants or fancy marketing terms for what we were trying to accomplish at the time. We just knew that we needed to improve distributed code development.  So we started first by architecting and then releasing Subversion as open source as it improved on the weaknesses of CVS for collaborative software configuration management over the web. But something more was needed: The open source development practices and objectives outlined in the bullets above needed to be codified into a complete a set of agile application lifecycle management (ALM) tools and development processes, which could then be applied to provide commercial benefits to enterprises around the globe. Since launching cloud development in 1999, the global adoption of these principles within enterprises has been impressive; so much so, that I can now knock on the office doors of developers and CIOs alike and simply say “agile development in the cloud” and we commence an immediate, constructive dialogue on needs and vision.  

Collective Cloud Intelligence and DevOps—an Example
Imagine using open source, agile, and cloud techniques to create hundreds of interconnected applications built by thousands of distributed developers at > 100 global providers. Imagine further an interconnected SOA architecture where applications and services are continuously delivered in minutes to weeks to months both elastically and manually into both public and private data centers.

This is not just a dream; this is a reality. I’ll describe one example: Deutsche Post in Germany, which has achieved this by applying the techniques discussed above.

Prior to employing a development cloud, the Deutsche Post development practices were similar to the industry’s current disparate and fractured state as described above. Too much time was spent on tasks that added little apparent business value, and often only impeded business agility. For some older applications, locating the original source code or understanding the rationale for certain programming decisions were already time-consuming tasks. There were large differences in terms of process and release automation maturity across the organization. Release processes, manual and error-prone, were lacking a common standard.  Many the Deutsche Post’s 300 development and maintenance projects were using its own source code and artifact repositories, development methodologies, and software tools. There was a large degree of redundancy, mainly in storage locations of documentation, and lack of cohesive processes for application delivery and tracking of software quality metrics. Development and maintenance errors often only became apparent in integration, sometimes with errors being fixed during acceptance tests or even only once the code had moved to production.

Deutsche Post rectified this by putting in place a cloud-based development platform that utilized the guiding principles I outlined above: Transparency, collective intelligence, process codification, cloud, openness, and community architecture. Utilizing CollabNet TeamForge as its core ALM integration platform, Deutsche Post built its lifecycle management platform to integrate the aspects of resource management, source code management, and continuous integration and delivery from code to “build and test” to actual “release and application deployment.”

In particular, integration with HP Quality Center, HP Operations Orchestration, and HP Server Automation software was seen as key.  Also important was the ability to interface with Hudson, Eclipse, and Subversion.  Server and cloud provisioning were enabled using elastic and manual provisioning to both Beanstalk and Amazon EC2, as well as to infrastructure provider cloud T-Systems. The above infrastructure was enabled across their enterprise for scalability to greater than 5,000 users codifying a variety of development processes from waterfall to agile to Scrum into the platform.

The Business Value of Development in the Cloud
The business value of cloud-based development through implementing continuous delivery and DevOps can be large. In the Deutsche Post example above, the organization reduced its IT ops budget by 20 percent, decreased time to market by up to 40 percent, improved developer productivity by up to 30 percent, and achieved ISO/IEC compliance. Specific gains were incurred through:

• Transparency and traceability of changes to IT systems
• Increase in quality of applications in operation, including improving the transitions from development to maintenance and operation
• Standardization of tools, processes, and information
• Efficient management of different service providers
• Acceleration in processes, including operations release cycle improvement up to 50 percent
• Simplification and standardization of project setups
• Reducing costs, for example, by replacement of individual solutions
• Central provisioning of IT systems for developers and testers during the application lifecycle
• Reduction of the used software portfolio across all IT systems

Best of all, is the time to business and technical value. Because the principles set forth above have been codified into an operational platform, Deutsche Post is able to onboard new teams and see benefits in fewer than ten days for each project.  Calculate this business return by multiplying the percent productivity gains and cost reductions mentioned above for each of the thousands of developers, and hundreds of projects and applications, across both the development and operations spectrum. It’s impressive.

Summary
In a nutshell, here’s what you need to know about the development cloud.  Simply put, agile, and public cloud platforms have been great for delivering value to individual product development teams. To scale these benefits to the enterprise, you should follow the holistic approach incorporating the community architecture and guiding principles I laid out above.

To make the most of cloud development, look for ways to integrate departments, share information, and create repeatable, collaborative processes that scale from the workgroup to the enterprise and from development through production. If you do, the gains can be big for you and your organization.