Not Every Restart Is Free
Sidecar upgrade sounds like infrastructure. For your pods it means: restart. For your clients it means: it depends.
- Date
- Topic
- Lifecycle
- Sources
- 2
- Reading time
- 2 min
- Version
- Translation
After the canary upgrade from report #20, workloads switch revisions at the pod restart. With that, sooner or later the entire fleet rolls through once. The question that matters: what happens to the requests that are running right now?
Figure as text
Timeline of a pod shutdown from stop signal through the drain phase to exit, with three request fates: a short HTTP request served to completion, a new request during drain cleanly rejected because the endpoint is already out of rotation, and a gRPC stream or WebSocket cut at exit, no drain window is long enough. Below, the pattern: test once under load instead of assuming, and the note that long-lived clients need tested reconnect logic.
The sequence at shutdown: the pod gets the stop signal. The proxy enters the drain phase, accepts nothing new and lets running requests breathe out. Then the exit. Whether clients notice any of it is decided by the interplay around it: readiness flips, the endpoint leaves the rotation, the rolling update keeps enough capacity available in parallel. When these gears mesh, a short HTTP request is a non-event.
The special case is long-lived connections. A gRPC stream that stays open for hours, a WebSocket in continuous operation: those survive no drain window, however patiently it is configured. At some point the exit cuts the connection. These clients need reconnect logic, and tested reconnect logic at that, not the one in the legacy module that everyone hopes still works.
The pattern behind it: test drain behavior under load once instead of assuming. Terminate a pod under real traffic, measure error rates and dropped connections, watch readiness and the rolling update while it happens. Proof instead of gut feeling, the line of this series since report #1.
For decision makers this means: every sidecar upgrade is a restart of the entire fleet. Whether that becomes a non-event or an incident with customer contact is settled before anyone starts the rollout. The only question is whether you proved it beforehand or only find out during the rollout.
If you have never seen your drain behavior under load, you have not seen it yet.
Which long-lived connections run through your mesh, and when did someone last test the reconnect under load?
