An Agile Approach to Retail: A Data-Intrinsic Development Case Study

Traditional software development methodologies (SDMs), also referred to as software development processes, are being replaced by light or agile SDMs. Agile SDMs are characterized by iterative development, continuous code integration, and the ability to handle changing business requirements [1].

Extreme Programming (XP) is the most well-known agile methodology. XP is based on a series of concepts that include: having the business customer on-site, pair programming, collective code ownership, continuous code integration, small releases, designing tests before writing code, standup meetings, refactoring, and 40-hour work weeks [2].

Disciplined agile software delivery processes are Dynamic System Development Method (DSDM), Open Unified Process (OpenUP), and agile instantiations of the IBM Rational Unified Process (RUP) process framework. All of these methodologies function differently from traditional SDMs and help organizations meet the challenges of the digital economy [1].

This paper covers the background of the project, approach taken, benefits gained, challenges and conclusions.

2. Agile Software Development Methodologies
 

Agile methodologies are used to produce higher quality software in a shorter period of time. Agile methodologies were developed to streamline the development process and remove barriers to accepting business requirement changes during the development process. Agile methodologies do not require that business requirements and design details be locked in for the duration of development [3]. Agile SDMs share several features including prototyping, iterative development, and minimal documentation [4].

2.1. Extreme Programming (XP)

XP is a collection of practices for software construction, include refactoring, test-first design, pair programming, on-site-customer, continuous integration, whole teams, and collective ownership.

XP Practices

• Test driven development:
• Focusing on Thinking about what the code should do before doing
  • High Quality and defect minimization
  • Regression test baseline
  • Create confidence in development team

• Refactoring:
  • Improve quality, reducing code entropy
  • Re-organize design, to lessen the cost of change to Simple Design:
  • Avoid paying the cost of potentially useful, but not business critical feature.
  • The design evolves according to client business requirements

• Pair Programming
  • High Quality standards through constant Code Review
  • Cross Training of the whole team

2.2. SCRUM

The focus of Scrum is project leadership and requirements management. Scrum proponents claim that it is a process framework, but if so then it is a sparse and incomplete one at best. A more accurate description is that it is a high-level lifecycle for construction iterations (what Scrum calls “sprints”), see Figure 1, and several practices such as daily stand-up “Scrum” meeting, product owner, product backlog, iteration/sprint planning, and potentially shippable software.

Figure 1 Scrum Construction Lifecycle

The above figure has been referenced from http://www.ambysoft.com/essays/agileLifecycle.html

Scrum Approach

• Accept:

o This is a complicated, unpredictable world and development environment.

o You can't predict or definitively plan what you will deliver, when you will deliver it, and what the quality and cost will be.

• Assume:

o You can estimate, and then negotiate according to various risks as you proceed.

• Understand:

o Following any cookbook approach won't improve the definition of "best", and will only hinder appropriate responsiveness to the complexity and unpredictability.

o the delivery will be the best possible given the circumstances

For implementing scrum visit www.implementingscrum.com for further details.

2.3. Agile Modeling (AM)

AM is a collection of practices for light-weight modeling and documentation, including requirements envisioning, executable specifications, active stakeholder participation, prioritized requirements, and prove it with code. See www.agilemodeling.com for details.

2.4. Agile Data (AD)

AD is a collection of practices for database development, including agile data modeling, database testing, database refactoring, and continuous database integration. See www.agiledata.org for details.

2.5. Rational Unified Process (RUP)

RUP is a comprehensive process framework for iterative software delivery which can be instantiated anywhere from a very agile form to a very traditional form as your situation warrants [6]. RUP practices include risk-value lifecycle, whole team, test-driven development (TDD), and business process. See http://www-01.ibm.com/software/awdtools/rup for details.

2.6. Open Unified Process (OUP)

OpenUP combines and extends practices from Scrum, XP, AM and RUP for co-located agile teams which are building business applications. OpenUP practices include whole team, daily standup meeting, prioritized work items, risk-driven lifecycle, TDD, active stakeholder participation, and continuous integration. See www.eclipse.org for details.

3. Agile Approach

Agile Approach has to be tailored to the needs of the project one is working upon. Agile process adopted in this project was culmination of various precepts pronounced in various agile SDMs.

Various precepts applied in the current work with respected to Agile SDM were as follows:

• Requirements Envisioning
• Redefining Stake Holder Participation
• Incorporation of Late Requirements
• Work Prioritization
• Continuous Integration

   

• Pair Programming
• Refactoring
• Cross Training of whole team
• Applying TDD
• Technical Excellence

The above approach is explained in detail with all due examples in the subsequent case study section of this paper.

4. Case Study: Traditional to Agile Approach in Data Intrinsic Development

A retailing client made their first foray into ERP implementation by implementing an Oracle packaged solution called Oracle Retail. This program has been aimed to implement various modules of the Oracle Retail suite formerly called Retek.

The Oracle Retail Suite had to form the technological backbone of their retail venture. The process of this retailer was quite ad-hoc in nature. The IT infrastructure required to make this had to be robust and scalable to meet the dynamic changing requirements for the customer. This project aimed to transform and streamline the business and IT processes for this retailer in lines with world class best practices in the Retail industry

Retail is all about business agility. Requirements change on the fly. To beat the competition a retailer has to change the dynamics of its organization at rapid pace. This requires systems and processes to be agile. IT has to be an enabler for this happen. This in turn requires IT systems to follow sound agile process to deliver value to the customer in the minimum time possible.

Retail being a highly margin sensitive area, the retailer has to capture lot of data to optimize its pricing engine to get the best value to the customer and in turn protect its margins. This requires huge data churning within the retail organization. This required defining and proving the agile processes work in packaged implementation. Moreover, the sound agile processes adopted during the implementation become the sound backbone for the retailer for future.

This section details the various facets of agile software development applied for this retailer. The tenets of agile development have been put to the test and the experience gained from this has been penned down in the paragraphs below.

Requirements Envisioning

Ascendant Methodology was adopted to do requirements envisioning. During the inception of the project a high-level requirements modeling was done to understand the scope and identify the initial stack of requirements. This project being a packaged implementation, it was critical to understand the fitment of the customer requirements with the product. Any deviations were duly captured and communicated to the customer as a gap. This also made the implementation team aware of the gaps coming forward beforehand and just in time.

Redefining Stakeholder Participation

An agile approach in a traditional environment is really tough. People are not willing to understand the agile approach as such. Customers are not fully aware of the approach and have no willingness to hear it as well. The approach taken internally was to satisfy our customer as functional consultants using an agile approach. An Agile approach was to deliver to internal customers of the project team. A stake holder for the development team is the internal team which certifies the product before releasing to the external stakeholder.

Incorporation of Late Requirements

Initial requirements were prototyped. The design was envisaged keeping in mind the changes coming late in the project to be incorporated. Each report was driven by an Oracle Database View. Report were predefined by customer in ms excel templates. These templates were uploaded in the database. Any additional column required a change in template and subsequent upload a CLOB database object type. Since each report was based of database view it was just matter of adding the additional column in the view from the table already in use in the view definition.

Work Prioritization

Changing the stakeholder definition to an internal product owner customer helped the team to prioritize the work. The entire team was co-located and any suggestions on work priority were discussed right at the moment to decide team’s future course of action. Real stakeholders were intimated on the details of the work scheduled for the team. Communication to real stakeholders was done only after having internal stakeholder approval.

Continuous Integration

Continuous integration was one of the key approaches adopted in the data intrinsic development. In all instances the code was deployed based on a few iterations of the develop, build and test cycle. The development was done based on requirements provided by the customer. The requirements were then prioritized. Based on these prioritizations the development would work with iterations of development. The end product was continuously integrated with the bigger system to ensue this integration did not break the larger system. This resulted in smooth transition during the movement of products in a production environment.

There was an instance wherein an application was developed in the corporate environment using an intranet network. This application was supposed to be deployed at a remote location for the customer. This application was working perfectly fine when the server was accessed using the local area network (LAN). But the application just was not working when deployed in the remote location.

The connectivity to the server was not appropriate for certain protocols required in the application. This was a serious issue since without opening up this protocol in all the routers to the server this application wouldn’t work. The design was re-organized to circumvent this in a two step process. This design was also thought through with the customer in mind. The first step was making a direct database entry of the data to be transported from a local client to the server. The second step was the creation of the file at the server side using database server file utilities. These two steps isolated the protocol error occurring in the application. The cost of change was quite a bit less since this kept the other steps in the application intact and re-designed a portion of the application process.

During one of the demonstrations of code developed during the project the agile approach was quite handy. Two team members from the team were executing the demonstration to the customer. One of the programmers was displaying the features of the code and performing the initial tests for the customer on the developed application. This demonstrator was connected to the projector for the entire audience. Another programmer was sitting just beside and noting the comments of the audience. There were a couple of enhancements requested by the customer during this session. While the first demonstrator was demonstrating the application, the second developer already coded these new features in the code and deployed the same code during the recess break of the session. After the recess break once more the application walkthrough was conducted. This walkthrough resulted a huge success of maximum of changes requested by the customer successfully implemented and demonstrated within a very short duration.

Pair Programming

There was an instance in the project where we had to deploy an unknown application. This definitely required some initial training to get the job done. The customer arranged for online CBT training but this did not help. We did form a two member paired team to get the job done. There was one programmer doing the typing effort and another person religiously followed the application deployment guideline to guide the typist. This combined development effort paid off and we were successfully able to implement the application in very short duration. As a side note I would emphasize that this required strong talent, effort and attitude to get the job done. Pairing made it possible.

Cross Training of whole team

All the members in the team were provided an opportunity to work within other modules of the project. The whole team was collocated in a single venue. Team sitting in a single location actually helped in various ways. Discussions between team members of various modules by and large happened in the same room. This resulted in each and every individual having a fair amount of idea on each module being implemented.

As in IBM servers, it applies to humans as well. Not all the time is each individual fully engaged. The IBM virtualization concept utilizes spare processing power from servers in the stack which are not in use. Applying the same concept in the co-located team, it helped in filling any immediate requirement from a module which would require some extra effort for any dynamic requirement.

This in turn also gave some amount of knowledge to the cross team member who approached to solve the issue.

Applying Test Driven Development

TEST DRIVEN DEVELOPMENT (TDD) LIFE CYCLE

The below diagram explains the TDD process lifecycle.

See http://www.agiledata.org/essays/tdd.html for further details in TDD processes.

Add a test

In test-driven development, each new feature begins with writing a test. This test must inevitably fail because it is written before the feature has been implemented. In order to write a test, the developer must understand the specification and the requirements of the feature clearly.

An instance of a test in a the case study:

In development of a retail application for financial accounting, user stories of this application were beforehand given to the developer. In all the developments the developer was provided with use cases and stories to cover requirements and exception conditions. This made developers aware of the requirement before writing the code.

The requirement was to develop an application that sends data on a daily basis of certain attributes and thereafter updating of a flag at the subsriber end. The test condition was written by the developer stating the attributes of the database required to pull the data from. If any of the columns required for a certain table do not exist then this test condition will fail. The code had to be rewritten to take the appropriate column from the table for the test to pass.

Run all tests and see if the new one fails

This validates that the test harness is working correctly and that the new test does not mistakenly pass without requiring any new code. The new test should also fail for the expected reason. This step tests the test itself, in the negative: it rules out the possibility that the new test will always pass, and therefore be worthless.

Write some code

The next step is to write some code that will cause the test to pass. The new code written at this stage will not be perfect and may, for example, pass the test in an inelegant way. That is acceptable because later steps will improve and hone it. It is important that the code written is only designed to pass the test; no further (and therefore untested) functionality should be predicted and 'allowed for' at any stage.

Run the automated tests and see them succeed

If all test cases now pass, the developer can be confident that the code meets all the tested requirements. This is a good point from which to begin the final step of the cycle.

Refactor code

Now the code can be cleaned up as necessary. By re-running the test cases, the developer can be confident that refactoring is not damaging any existing functionality. The concept of removing duplication is an important aspect of any software design. In this case, however, it also applies to removing any duplication between the test code and the production code.

Repeat

Starting with another new test, the cycle is then repeated to push forward the functionality. The size of the steps can be as small as the developer likes, or get larger if s/he feels more confident. If the code written to satisfy a test does not fairly quickly do so, then the step-size may have been too big, and maybe the smaller testable steps should be used instead.

The first step is to quickly add a test, basically just enough code to fail. Next you run your tests, often the complete test suite although for sake of speed you may decide to run only a subset, to ensure that the new test does in fact fail. You then update your functional code to make it pass the new tests. The fourth step is to run your tests again. If they fail you need to update your functional code and retest. Once the tests pass the next step is to start over (you may first need to refactor any duplication out of your design as needed, turning TFD into TDD).

I like to describe TDD with this simple formula:

TDD = Refactoring + TFD. [5]

This Test Driven Development approach was followed across on new developments done in the

project.

Technical Excellence

The customer had proposed for development of 16 reports as part of SOW at an implementation project at a large India based retailer. Since the reports were not classified by complexity the customer proposed very complex reports for development.

The challenges for the report development

1. Each report was the culmination of three to four medium complex reports
2. Excel output was required for the reports which was not in purview of development effort/methodology
3. Huge Technical Challenges

Based on the requirements the report development was started by developing the skeleton or framework for one of the reports. Brainstorming was conducted on technical challenges of the report and work for the same started. The working report was ready in a day. There were issues of performance in the report as far as the data display in the screen was considered. The data was displayed in the spreadsheet on a cell by cell basis. Oracle’s Webutil technology was leveraged to write data in the excel sheet.

The following steps were taken to foresee any bottlenecks and combat well in advance

• Since each report was based on a database view, SQL tuning the underlying view database query was indispensable.
• Server Statistics reports collected on a daily basis checked the server health and applied appropriate checks if necessary
• Database performance was continuously monitored by modifying database parameters
• Collecting the data base statistics reports.
• Each SQL session and application session was traced. Oracle database utilities were used to analyze the performance impact of session parameters

This approach of continuous integration and collective ownership as a team was adopted to benchmark the technical approach in an agile way.

5. Benefits Gained

By following a thoroughly agile approach to software development for data intrinsic development following was gained:

• Design and Build of reports were prototyped resulting in creation of working software before discussing requirements further with customer
• Effort spent on creating this working software was restricted to two days only
• New alternatives/suggestions were advised to customer on first meeting itself
• Realistic expectations were set for the customer from day one based on the technical limitations of the development
• Functional fitment done well in advance because of requirements envisioning
• Ability to take any additional touch-ups on the code as along as the framework of the development does not change
• Customer was very satisfied on the approach of creating prototype in initial phase
• Requirements late in the project were easily incorporated
• The working reports were used by the customer for extensive data analysis from their end.
• They used the reports to further verify the data in their system
• On the fly change during presentation using Pair Demo as part of continuous integration was just excellent and was highly appreciated by the customer
• Pair Programming resulted in huge customer satisfaction
• Quick Results
• High Quality standards were set through refactoring codes
• Good Clean code that works
• Agility in the Software development process
• Huge confidence in the development team

6. Challenges

There were many challenges faced in journey of applying agile-tional approach to development

It was difficult to make the team agree on the agile approach since their school of thought was more towards a waterfall model

Getting the buy in from the Project Management was tough and hence a workaround stakeholder idea was devised

Technical Challenges arose like publishing an excel report from Oracle 10g server

Performance of the reports in general was an issue and this was dealt with by continuous monitoring of the database and application

Regression Testing of deliverables upon incorporating late feedback maintained quality

7. Conclusion

Software development can be done in an agile way bringing delight to the customer as an end result. For this retail operation, it provided quick turnaround time to deliver quality product to the customer. Willingness from the all stakeholders is indispensable for successful agile methodology implementation. Requirements changes late in the game can also be incorporated with agile SDM. The customer is fully aware of the end product getting delivered because they participate in building it.

Agility is more of perspective. It’s really just a mindset. It’s about adaptability to changing business requirements. Based on the points highlighted in this paper, and supporting points given, it could be very well stated that agile SDM can be adopted in an organization whilst providing huge benefits and requiring few cultural changes that will greatly impact the implementation.

8. References

1. Boehm, B. Turner, R. Management challenges to implement agile processes in traditional development organizations. IEEE Software. 22(5), 30-40. 2005.

2. Theunissen, W., Boake, A., Kourie, D. In search of the sweet spot: Agile open collaborative corporate software development. Proceedings of the 2005 Annual Research Conference of the South African Institute of Computer Scientists and information Technologists on IT Research in Developing Countries. White River, South Africa. 268-277.

3. Lindstrom, L. Jeffries, R. Extreme programming and agile software development methodologies. Information Systems Management. 21(13), 41-53. 2005.

4. Holmstrom, H., Fitzgerald, B., Agerfalk, P., Conchuir, E. Agile practices reduce distance in global software development. Information Systems Development. 23(3), 7-18. 2006.

5. Agile Database Techniques by Scott Ambler

6. Kroll, P. and MacIsaac, B. (2006). Agility and Discipline Made Easy: Practices from OpenUP and RUP. Boston: Addison-Wesley