Plans…..Too Sacred

It is baffling how sacred we still treat formal plans in enterprise software projects.

On average, 70% of projects miss the plan on some dimension. Yet, we try hard to conform to each one we create. Teams endure a lot of unnatural contortions to preserve the sanctity of those plans.

When projects go south, the consensus takeaway always is to make those plans more sacred, i.e., more thorough and more scientific planning.

That is the exact opposite of what the remedy should be.

Responding to change?

17 self proclaimed “organizational anarchists” met at Snowbird Ski resort in February 2001 and forever changed the world of software. They wrote the Agile Manifesto to uncover “better ways of developing software”. One of the ways was by valuing “responding to change over following a plan”.

Today the enterprise software industry follows Agile with an almost religious fervor. When we struggle to get it right, we invent new methods (e.g., SAFe). Or come up with new ways to brand our efforts(e.g., Hybrid Agile).

Before Agile, we had Gantt charts. Now, we have Gantt charts with sprints on them. And we add copious buffers to cover for plan variance. We ar e so scared of missing the deadline that we willingly accept being late. That doesn’t sound like Agile to me.

Methods have trumped the Principles. We fret obsessively over sprints, ceremonies, and retrospectives. And we have forgotten about getting better at responding to change.

The Planning Fallacy

Over-optimism in plans goes beyond software. It is a human cognitive bias. The Nobel laureate, Daniel Kahneman, named it as the Planning Fallacy.

Per Kahneman, we over-index on the causal role of our skills and things we control. We ignore causal interactions from outside the project and existential reality we don’t control.

Making deadline promises based on software estimates done during the most uncertain phases of the project is an example of Planning Fallacy. And making such planned deadlines sacrosanct should even count as a form of self-flagellation.

Like with buffers, team incentives stem from the risk you prioritize. Fear of arbitrary deadlines drawn well in advance duly tramples over any creative urge to build and do the right thing for customers.

A plan too sacred is not only painful to execute, it also tamps down creativity. Whenever innovative software is proving to be elusive, a straight-jacket of rigid planning is often the reason.

It seems like Agile guys knew about that too, when they wrote that they valued “working software over comprehensive documentation.”

Learning from “Working Software”

There is a clear trajectory to enterprise software with the cloud. Commodity technology becomes reusable open-source software (e.g., Javascript frameworks, ML frameworks). Templates become products to buy (e.g., Cloud App Ecosystems). Products become services to rent (e.g., Cloud Services).

In the reuse-buy-rent (RBR) first paradigm, only unique software and new recipes create differentiated customer value. The currency for creating new software recipes is “working software”.

However, “working software” as a measure of progress is easier said than done in an enterprise environment. Leave aside executives and managers; solving only immediate problems with an ambiguous vision can be paralyzing even for many developers.

It could be unintuitive. But “working software” enables directions that uncover possibilities that planning can’t identify.

Because staying flexible on what you will ship in 3–6 months prevents underestimating what is possible in the long term. The clear next step to ship “working software” in the next 2–4 weeks prevents overestimating what is possible in the short term.

Operating from “working software” is like learning to walk in the fog. We plan for what we can see. Once you execute that plan, more of the path shows up. That allows planning for that next course.

What is opposite of a Sacred Plan?

We need planning that can help respond to change better, mitigate the planning fallacy, and leverages “working software” as a learning device.

An unfavorable project outcome should not trigger a longer planning cycle to get it right the next time. Because investing more in planning makes plans more sacred.

Instead, we need to move in the opposite direction. A shorter planning cycle represents the single most significant opportunity to improve the performance of enterprise software projects.

If you have a six-month planning cycle, a three-month one is better. A month is better than three. Each organization should optimize its software planning cycles, but the best ones ship daily.

Operationally, a short plan with “working software” as an ideal reduces the batch size of software development. Small batches increase the frequency of customer and user feedback. They allow developers to keep up pace with cloud software. Iterative discovery of customer value and staying on top of the RBR landscape is the source of real innovation in modern software — not grandiose visions.

We have learned in the 20 years of Agile that it is impossible to make developers Agile when everything else around them isn’t. Therefore, the age of software requires everyone to think like a software person.

Delivering on shorter plans gets everyone working in that direction.

Cover Art by Kelly Sikkema on Unsplash

Originally published at https://www.amsidhu.com on May 18, 2021.