Waterfall or Agile - Why not both?

In 2007, I was introduced to the idea of agile project management as part of my Masters’ degree. It made no sense. The company I worked for constructed multi-billion-dollar projects for the energy sector. From this starting point, it was difficult – if not impossible – to understand how a project would be successful without up-front design, sign off of requirements, careful change control and significant amounts of governance related to scope, cost, and timeline. Simply put, it wouldn’t work given the constraints we were operating under. For example, Design had to be signed off by the client by date A, so that the fabrication yard in Korea could build that component by date B, it could arrive in Yemen by date C, be installed by date D, gas production could start by date E, and the client could meet their commercial commitments to sell gas to consumers by date F. In this scenario, missing any one of those dates would put the commercial outcomes at risk, and result in liquidated damages – fines for being late – being imposed on the company I worked for. Agile practices sounded completely infeasible for the world of construction projects that I was immersed in.

Fast forward to 2012. I’m in a different company in the energy industry. For the first time, I’m a program manager working on a software project for internal customers. Tasked with building software for hydrocarbon analysis and reporting, I set off on the path that I know well: define requirements, get sign off, hand them to the developers, test that the product meets the specifications, hand it off to the customers. But when the customers received the software, they weren’t thrilled. “Could we add…”, “this isn’t exactly what I meant when I said…”, “No – this won’t meet my needs.” I asked myself what had gone wrong – the unhappy customers were the same people who’d agreed to requirements, specifications and mock designs. Instead of celebrating a successful ‘go-live’, it felt like the team would have to go back and rework large sections of the product to make it useful. The waterfall approach that I’d relied on previously, had come up short.

After discussing with the developers, we decided to experiment with an agile approach for phase 2 of this project. Leadership resisted. All of our software development practices were built for a stage-gated waterfall methodology. Agile wasn’t allowed. Off the record, several leaders were willing to acknowledge that things didn’t always work well in waterfall, but the sign off stage gates afforded them necessary protection against the customer, if (when) the customer wasn’t happy with the end result. We were using requirements and specification documents, coupled with governance stage gates to write contracts with our colleagues. Underpinning the need for contracts was a lack of trust based on years of tension and perceived poor outcomes. Our team knew it could be better than this, so we decided to experiment with agile practices, specifically Scrum, while maintaining a façade of waterfall.

A project manager with experience of scrum joined our team. We all had to adapt. Our understanding of individual accountabilities and boundaries, was transformed into common goals that we worked on together. We informally demonstrated the product to customers at fortnightly intervals, and collated feedback to help us plan changes or next steps. Trust grew. Respect increased. Some of the customers began to reach out with suggestions or guidance without prompting. Quietly, we’d started to prove that transparency, inspection and adaption – core to empiricism – were helping us build a more useful product. However, our ability to be transparent was limited to the team, and a handful of customers. For our leadership and the rest of the organization, we’d created an effective set of waterfall documents: project status reports, stage gate reviews, cost and budget reports and Gannt charts. For the remainder of this program, we stealthily grew our skills around scrum and agility. We had better outcomes, happier customers and a more engaged team. Yet, we had more work to do to convince our leaders that this approach could be more widely adopted – that will be a story for another time!

What I learned between 2007 and 2012 was that both waterfall and agile approaches work. The best methodology depends on the problem you’re trying to solve. In the case of the construction projects, there were established best practices across the industry, there were blueprints that my company used for all similar projects, there was a pattern in place that we knew would be acceptable to our client ahead of time. Before beginning, we knew what a good outcome would look like. Whereas in the software project, the problem was not well defined. The customers themselves did not have a coherent and aligned perspective on what outcome they wanted. This was in the complex domain of product development, which is far better suited to agile methods that permit an iterative and experimental approach. Sometimes we see agile and waterfall pitted against each other, as if one is inherently better than the other. As agile practitioners we need to know the benefits and drawbacks of both approaches to help our clients solve their business problems in the most effective way, rather than blindly force-fitting our preferred framework to the problem.