Agile–What’s the problem?

Within my social circle there has been a lot of discussion around agile, scrum and waterfall software development lifecycle techniques (SDLC). The basic issue is that the grass roots feedback coming from sales and delivery is that the customer is asking for agile based delivery and more over, scrum based delivery. Anecdotal evidence suggests that the companies may miss out on lucrative projects through not offering a scrum approach to delivery. Now personally I’ve not seen any customers reject proposals by not offering a scrum approach however that’s not to say that it hasn’t happened. So the question has to be asked why I don’t subscribe to a scrum approach in all cases?

Well the first and foremost issue is that many of our customers request either a fixed price approach or a fixed date approach neither of which fits well with a scrum approach to the SDLC. Fixed price depends on a full understanding of the requirements and a full set of designs before committing to a price, or a significant proportion of the price is contained within contingency to accommodate the risk associated with not knowing the detail behind each requirement. Likewise an understanding of the complexities of interfacing to applications within and without the business domain is required to develop confidence in your commitment. The fixed date approach also requires a set of, and to quote the (ahem) USA philosopher Donald Rumsfeld, ‘known, knowns’. Even Ken Schwaber in his book ‘Agile Project Management With Scrum’, realized that the scrum approach ‘had no silver bullet’ for fixed price/fixed date delivery, going so far as to state that the approach would require ‘adding a waterfall phase to the front of the scrum methodology’, anathema to most scrum advocates.

Steve McConnell in his book ‘Software Estimation’ differentiates between estimates, targets and commitments when estimating software development projects. He states that ‘Businesses have important reasons to establish targets independent of software estimates…While a target is a description of a desirable business objective, a commitment is a promise to deliver defined functionality at a specific level of quality by a certain date’. He goes on to state that ‘a commitment can be the same as the estimate or it can be more aggressive or more conservative than the estimate’. So what does this mean for SDLC?

Customers who are looking for certainty in what is generally an uncertain process want a commitment, either financially in the form of fixed price or schedule, hitting a fixed date release. Since I base pricing on my ability to estimate the delivery of the project on ‘known, knowns’ through a structured approach using models and spread sheets I have to understand and communicate the approach and its impact on the SDLC in a way in which the customer will understand. Steve Resnick et al, in their book ‘Professional Scrum with Team Foundation Server 2010’, make a distinction between waterfall, agile (which they call MSF after the Microsoft Solution Framework) and scrum:

• Waterfall – Scheduling is predictive. Using a known team and known technology, an experienced team can predict the duration of each phase and task. This method doesn’t respond well to slippage, as dependencies among tasks and phases are often very complex.
  
• MSF – Scheduling is predictive, as in the waterfall method. However, because MSF is iterative, with more frequent releases, schedule slippage is more manageable. Subsequent releases can add or remove features to react prior impact.
  
• Scrum – Scheduling is empirical. Work is scheduled based on the scrum team’s velocity. Estimation becomes more accurate with each successive sprint, based on actual work completed. Scheduling is very reliable because of the fixed-duration sprints. The scope is less reliable because features will move in and out of sprints and releases to accommodate the fixed schedule.

So what can we take away from these points. Well the first two approaches, waterfall and agile, have a very structured approach to gathering and engineering requirements therefore we can be very predictive around development of the estimate and the delivery schedule at a cost of being less reactive and flexible in adding additional requirements or making a change in the priority of the requirements. The third approach, scrum, has reduced confidence in predicting a schedule due to the nature of the backlog/sprint technique. It is extremely difficult to estimate the initial duration of the project when the points associated with delivery of the requirements within the backlog is based upon velocity, which is based upon historical evidence of delivery of software into the SDLC. This poses difficult questions around how you make a commitment to deliver requirements or how you make an accurate estimate for a fixed price/fixed date SDLC using scrum. So based upon this information, it looks like an agile approach would be the best at satisfying our customers requests to be flexible, yet agreeing to a commitment. I currently have an agile approach which works for me that develops the requirements and designs outside of the iteration using test driven development and continuous integration. This gives the best confidence in providing a price and a date to a customer and in making a delivery commitment to them.

At this point in the discussion I’d like to make in important point. Agile ≠ scrum. In fact for the best breakdown of myths regarding agile and scrum take a look at Eric Brechner’s book, ‘I.M. Wright’s Hard Code’, (2nd edition). He has many good points on this topic based upon his experience of the SDLC within Microsoft. Let’s take a look at his entry from March, 2006 entitled ‘The Agile Bullet; Enemy of the truth’:

• Myth #1: Agile = eXtreme programming (pair programming, scrum, test driven development, user stories or some other agile method).

Actually what he describes in this section is that agile is actually a collection of many different SDLC approaches. You have to ensure that you have the right approach for the right project. I have spent a lot of time developing methods and I feel confident that the agile approach I have adopted will accommodate my customers requested commitments.

• Myth #2: Agile methods can’t work for large groups.

I think that this is tosh, agile ≠ scrum. Agile doesn’t require that the customer product owner sits with the team 100% of the time writing user stories. I often work in a distributed fashion with more and more emphasis placed on an offshore delivery model using the India, Manila, China and Brazil. This means that it is impossible to have the product owner sit with the development team. Therefore an approach which works for this model is required. Agile provides the ability to develop requirements and designs outside of the iterations therefore it can accommodate large scale and distributed development teams.

• Myth #3: Agile methods can work for large groups.

Now I know what you’re thinking “how can he justify this statement when he’s just praised the approach for large groups?” Well, Eric has a good deal to say on this. ‘The agile philosophy values “customer collaboration over contract negotiation” and “responding to change over following a plan”…[customers] get touchy when millions of dollars are involved…[therefore] applying agile methods to large-scale projects requires you to be flexible and creative to deal with these issues.’ So the approach is to make a commitment to fixed price/fixed date delivery based upon an on going discussion with the customer to ensure that they are happy with the priority of the functionality being released from the SDLC and also to discuss or negotiate the impact of changes on the SDLC and the fixed price/fixed date delivery. This means having a team at the customer site working with the product owner and communicating to the team at large.

• Myth #4: Agile means no documentation.

Eric makes the point that ‘the agile philosophy values “working software over comprehensive documentation”’. So what does this really mean; No documentation? Of course not. Documentation should be seen as a deliverable and should be factored into any estimate and planned accordingly. Documentation makes the maintenance of the application possible and helps the organisation further develop the application in the future.

• Myth #5: Agile means no up-front design.

When making a commitment to a customer, even using the agile approach, a certain level of design documentation is required. How else do you work out the complexities of the application architecture, which patterns you may use, what software models you may select? Again the design will not emerge from developers ‘doing stuff’. The balance is to generate enough of a design to help with the estimate and to drive requirements engineering so that the backlog and iteration plan is as well developed as it can during the SDLC.

So all in all, we’ve looked at the reasons why I prefer agile as opposed to the scrum approach to the SDLC, we’ve looked at the differences between estimates, objectives and commitments and finally we’ve looked at some to the myths associated with the agile approach. Wrapping this entry up I think we can safely say that although scrum is an excellent approach if you are not making a commitment to your customer and if they can make a mutual choice between delivery of functionality or delivery to price/date. However, most of our customers have fixed budgets and a timeline to hit making scrum a difficult choice pushing us to a decision between waterfall and agile.

If you’d like to have a further discussion then please get in touch.

Technorati Tags: agile,scrum,waterfall,software Services,software development,SDLC,Microsoft,Team Foundation Server,TFS