And why Product Managers have a key role in fixing it
Its been said by many that Software is eating the world, but still time and time again, software projects are like werewolves, mythical beasts that look like ordinary humans but sometimes turn into dreadful monsters.
“Software” has become an essential requirement in any company; for some, “software” is the engine that runs the internals of the organization. In contrast, others build Software as a product and sell it. Today even companies who sell hardware find themselves relay more and more on Software, both as the glue and logic between components, the integration layer with other systems, and the human interaction part.
Still, even after 60+ years of software development, Software is one of the most problematic kinds of engineering fields, with unsolved problems and a source of many pains in the modern business world.
In this post, I’ll try to break down some of the core reasons and point out fundamental strategies to better tame this beast.
When we try to analyze the difference between software projects and projects in other engineering fields, several distinctions come to mind:
• Complexity
• Conformity
• Volatility (Changeability)
• Invisibility
The bathroom example
I will try to simplify those ideas with an example since I thought of a case while I was taking a shower, a shower it will be :)
Complexity
Software entities are more complex than any other human constructs, while it’s important to remember that complexity is an essential property of Software.
This complexity is even more visible when you consider that no two software entities are alike
when such a case occurs, we immediately remove one of the instances and use the other as a generalized routine.
Another aspect of complexity is the number of states that Software contains.
That freedom of states within Software is why we choose to use Software to solve those problems.
Furthermore, as software modules start to interact with each other, those states are rapidly growing in a none linear fashion.
Let us build a shower
The first step in our shower example is to choose pipes.
In the physical world, we would probably buy those from a third party; they will have size, length, and other properties.
They will have values for maximum pressure they can withhold and the amount of water per hour that they can run. An engineer will do the calculations, think about the options, but in the end, she will choose one instance and place an order.
For the software engineer, it will be much different; most of those properties will be set as parameters since we probably don’t know what will be the best value for each. Our software solution will include versus colors, materials, and shapes; for each of those, we will need to code
relationships and dependencies, by the time the Software represents the piles will be over we will probably have more configuration then actual code. If we’re lucky, we will be able to cover all of the relationships between the properties.
Conformity
Software engineers are not the only engineers who deal with complexity.
Let’s take Physics; for example, we can easily get lost in the complexity of the physical world, especially as we get deeper into the fundamentals and the particle levels.
Still, there is a big difference between those fields, as, within the area of Physics, the rules conform. Once we define theory and prove it, we can count on its behavior.
In Software, it doesn’t. Since each module was designed and coded by a different team, there is no simple way to build a theory that will predict, with a high degree of success, how does a specific piece of code will behave with a particular set of inputs.
Back to the shower
Now, that we order all of the parts we start and build it, a fundamental part of a shower is the flow of the water.
As an engineer, one will have to understand the water flow, since we’re building a shower on earth we can count on gravity to pull the water down towards the ground. Relying on that fundamental rule, we will probably put the shower-head up while we put the drain pipes will below.
Not so simple when we get to our software shower, since everything needs to be defined, the shower-head team might think that it would be much better if the water would move sideways.
The drains group might think that draining the water from above will yield maximum performance.
Every behavior aspect must be coordinated and predefined, and every underlying assumption must be communicated to all (if not coded as an infrastructure).
Volatility (Changeability)
Software, by design, is meant to be changed.
There is no other manufacturing area, not cars, buildings, digital computers or airplanes, that gets change request the moment the product reaches its customers.
Many times, recall of a batch of cars might cause massive loss to the car manufacturer, while software patches/updates/versions are always part of the software life cycle.
The software volatility is by design, as it needs to hide areas of unknowns. Sometimes the volatility goal is to enable the search of product-market fit. Other times the volatility meant to incorporate the different needs of different customers.
Software volatility can even serve as an easy way to extend the longevity of physical products by enabling manufacturers to adapt products to market changes that didn’t yet happen at the time of manufacturing.
Can I get a bath instead?
Our engineer finishes the shower, proud and happy she waits for the customer to see it and collect the fee for the creation.
How many of the customers will step in and after a second announce: “Mmmm, that looks nice, but I think I want a bathtub instead.”
In Software? Those kinds of requests are the norm.
Invisibility
Software is invisible; not only that Software is hidden, its complicated to visualize it.
When a software architect visualizes a software project, he typically ends up with multiple visualization layers:
• Entities structure and relations diagrams
• Data flow diagrams
• Control flow diagrams
• Volatility diagrams
• Modules diagrams
• Dependencies diagrams
• Etc
Those diagrams are not even planar, not even hierarchical, and many times cannot be composed together, one on top of the other.
The problem of invisibility is not only in planning the Software the right way, but it is also in the lack of most people to analyze what they see and understand if it’s good or bad.
As software projects tend to live long, keeping those visualizations up to date, or build them again after hundreds of engineers contributed millions of lines to the code base becomes an unrealistic task.
Strategies to fight back
Now that we’ve mapped those four differences (Complexity, Conformity, Volatility, Invisibility), let’s discuss some strategies to fight back.
Complexity
Its extremely easy to ask for everything, to define a solution with many aspects of freedom and configurations.
The price for this kind of approach is very high, and due to the none linear nature of complexity, the cost piles up rapidly.
Product Managers should work hard and make sure they filter out noise, defining the essence of the requirements, and understanding the minimal degree of change and freedom. Only limitations can ensure that the solution will have longevity.
Conformity
Building a long-lasting and easily integrated software solution relays on the ability of everyone to understand the system rules and assumptions.
Systems should be simple, so that every developer, junior or senior, can explain it with ease. The primary role of the software architect is to layout the solution vision, then communicate it to everyone. Constant and clear communication between the Solutions Architect, the Product Manager, and the Customers is a must if we’re to define the ground rules of the system.
As the software changes, it’s essential to follow up on those assumptions, update them, and change the system as soon as those old assumptions become irrelevant.
Volatility
For most of the industry veterans the following five states in a product life cycle adoption will be familiar;
• Innovators
• Early Adopters
• Early Majority
• Late Majority
• Laggards
The critical phase, between Early Adopters and Early Majority (the chasm), is well defined for the marketing and product parts of the organization.
The software solution is not different; a software system that needs to support innovators and early adopters contains a high amount of volatility and uncertainty. Assuming that we can cross the chasm while the Software hasn’t been dramatically changed and refactor (sometimes even rewritten from the ground up) is dangerous. Trade-offs that “early adopters” might accept will not be acceptable by the “early majority.”
Here, once again, the Product Manager must identify that moment in the product life cycle, and a massive investment (defining, refactoring, and re-design) must occur in the software system.
Invisibility
There is no easy way around this vulnerability, and as far as I can see, the only way to address it is with great people.
One must insist on having the best Software Engineers around. Not only managers, architects and tech leads, each and every software engineer should be the best you can get. Important to say that I see qualification engineers, automation engineers, DevOps engineers, etc. as Software Engineers for that matter.
Once an organization size becomes significant, it’s even more critical as scaling software development organization is a hard task, especially if we must scale up due to the monolithic system, instead of scaling out (modular architecture).
Things I’ve left out
There are many other aspects I haven’t discuss that can help bring sanity into the twisted world of software development, quality first state of mind, modularity from the beginning, automation etc. Those points hold a significant role in the stability and longevity of the software system. Still, since those aspects are within the software organization control, I see them as part of bringing the best engineers one can find.
I want to read more!
The idea’s behind this post came from multiple resources. Once again I should recommend the “Mythical Man-Month” by Fred Brook (this time the second edition which includes more essays on software development).
“Crossing the Chasm” by Geoffrey Moore, which mainly discusses the life cycle of selling High-Tech products from a marketing point of view, is an essential view that we, as engineers, can learn a lot about the system we’re designing for each step.
