2024-03-12 · engineering · linux

Linux images in two days

A bank does not run "a Linux box". It runs a population. Different regions, different clouds, different auditors, one expectation: the next host should be boring.

When I joined the Linux platform side of HSBC the image cycle was ten days. Not because Red Hat is slow. Because the work was a relay of humans: a hardening decision in one team, a filesystem standard in another, a driver that needed a vendor email, a sign-off that lived in a queue, a test pass that started from scratch when any of those moved. Ten days was the sum of waiting, not the sum of compute.

What we changed

The job was to make a RHEL layer-1 image that could be consumed globally on public and private cloud without each region performing folk ritual. Ansible pipelines did the ritual. Low-latency Python services did the reporting. Maintenance packs became a product with a version, not a zip in a ticket.

Once the image is a product, the rest of the estate can be opinionated. The Ansible Linux Profile — OS, filesystem, cybersecurity, tooling — is the same idea one layer up. Standards that used to be slideware became code that fails closed. The licensing save (north of USD 30 million) is what happens when you stop paying for five ways to do one job.

The unromantic mechanics

  • Drift is the enemy, not the CVE. A CVE is a fact. Drift is how facts become incidents. Remediation without inventory is theatre.
  • Secrets are a pipeline. SSH keys and certificates that require a human to copy them will be copied badly. Vault, or you are lying to yourself about rotation.
  • Images are an API. If application teams cannot provision without a conversation, you do not have a platform. You have a guild.

We took the cycle from ten days to two. Consistency went up because the path got narrower, not because people typed faster. Resource utilisation went up because skilled engineers stopped being glue.

What two days still means

Two days is not CI for a container. It is still an operating system with firmware, filesystems, and a compliance story. Anyone who tells you "just use a distroless image" has not sat with a trading rack, a regulator, and a kernel module that has a name.

The right ambition is not zero days. It is a cycle whose steps you can name, time, and replay. If you cannot replay it, you do not own it.

I still think of platform engineering as a kindness to the future on-call. The image that boots the same way is a letter to that person.