Turbulence

TL;WR

If you’re moving your software development practices to Kubernetes, please don’t rush the packaging work (i.e. the charts, operators, and pipelines). It might seem that spending time here is a waste of effort, but rushing this part — or worse, working backwards from manually deployed customer environments — will cause expensive delays when you can least afford it.

A Dubious Analogy

In just about every business, the pressure to deliver can be immense. There are powerful incentives when deliveries go well, and there are serious consequences when they go wrong. The software business is no different, and sometimes a change of shipping strategies can lead to unpleasant surprises.

Here was a real chat from work on a Saturday morning, paraphrased for clarity.

DevOps Engineer: I’ve added success criteria to CDI-639.

Software Architect: Thanks. Shouldn’t you be gaming? Or golfing?

DevOps Engineer: I really should be. Nice day for it … but I heard the lambs screaming. The containerized DB lambs.

Software Architect: Hahaha … What are you doing to ease their pain?

DevOps Engineer: Unfortunately for them, their lives are meant to be short. I’m just trying to make their demise less traumatic … but seriously, I’m trying to decide how much chart work to do for <customer-x>. The DB charts need so much work, and we need to get the SQL feature activated. <customer-x’s> db chart has fallen behind, and now a bunch of things need to get in there, like <developer-y’s> port generation helper, and my pruning of duplicate chart overrides.

Software Architect: …

DevOps Engineer: Working backwards from the customer’s environment is turning into a nightmare. I guess this is a small version of site code. Someone needs to write a whitepaper on how allowing features to start at the customer side and work backwards creates chaos in the organization.

Software Architect: We have lots of custom … tip of the proverbial iceberg.

DevOps Engineer: I agree. This little lesson with Helm charts makes it clear how serious it could be on a larger scale. I can think of another metaphor that works best in one direction, and that’s digestion. Writing the whitepaper in those terms would make for a memorable one. Left-to-right metaphor. What’s on the left? Nutrients. What’s on the right? Healthy organisms (the good part), and waste products (an inevitable side-effect). Now, what happens when material from the right side makes it into the digestive system? Cannibalism and coprophagy.

Software Architect: Hahaha, fantastic analogy. A new trend in tech writing.

DevOps Engineer: Might be weird to call it a white paper. Maybe a brown paper?

A Slightly More Dignified Analogy

A less extreme analogy for force acting in opposing directions would be turbulence, hence the title of this piece. I find maintaining multiple target environments challenging, but this becomes daunting when each environment is a potential source of features for others.

I imagine this as a flow of information, where the normal flow would be inside-out (i.e. from developers out to customers). Where the trouble starts is when the information flows from outside in. Opposing forces create turbulence in an organization, and inevitably slow down delivery.

By all means get requirements from the customer, but make sure those requirements come through your pipelines from the left-hand side.

Solution

Spend more time working out how to make sure new features move in one direction through your deployment pipelines, including the packaging tools you use to deliver your software. You’ll be glad you did, and maybe your team will have some nicer Saturdays.