Certificates Age Faster Than Runbooks
The workload certificates in your mesh are short-lived and rotate automatically. For years, without anyone looking. That is exactly why nobody looks at the rest of the chain anymore either.
- Date
- Topic
- Certificates
- Sources
- 3
- Reading time
- 2 min
- Version
- Translation
Above the workload certificates hangs a chain of trust: a Root-CA, often an Intermediate below it, and only at the very bottom the short-lived part. The upper links do not rotate by themselves. Their expiry date is measured in production years, far beyond any team turnover.
Figure as text
The chain of trust above the workload certificates: Root-CA and Intermediate, which do not rotate by themselves, with long bars measured in years, and a red callout 'undocumented?' at the root; below them, many short-lived, automatically rotating workload certificates. Alongside, the one-minute visual check istioctl proxy-config secret (shows the rolled-out chain including validity dates) and the comparison for decision makers: a known CA expiry date is a plannable maintenance window, an unknown one a total outage you could see coming.
With the default install, Istio simply generates its own self-signed CA. Handy for getting started, and that is exactly where the problem begins: this CA does not show up in any of the organization’s certificate inventories, because it never went through an official process. Three questions as a test: who knows the expiry date? Where is the rotation plan? Has the rotation ever been rehearsed instead of only described?
The visual check takes one minute, and you know the tool from reports #8 and #12: istioctl proxy-config secret shows the rolled-out chain per workload including validity dates, also for the CA certificates above it.
What happens when the date approaches unnoticed can be worked out: under an expired chain there are no valid workload certificates anymore, and with STRICT from report #14, your entire service-to-service traffic depends on them.
For decision makers, it is a comparison of dates. A known CA expiry date is a maintenance window you put in the calendar years in advance. An unknown one is a total outage you could see coming, just without a date in the calendar.
The house does not suddenly catch fire here. It has an ignition timer, and it started running at install.
An assessment brings exactly these kinds of expiry dates to light, often as the most inconspicuous line with the biggest scare. If you do not know your date: feel free to reach out. And to everyone else: when did you last take a deliberate look at your mesh CA?
