Staging Lies
The sign-off came from staging, green across the board. Three weeks later a path burns in production that staging never saw.
- Date
- Topic
- Staging parity
- Sources
- 2
- Reading time
- 3 min
- Version
- Translation
I know this pattern from many setups: staging means “like production, only smaller”. Mostly only the second half of that sentence is true.
Figure as text
Parity table of staging versus production with five rows: injection (sidecar missing or outdated versus injected, current revision), policies (PERMISSIVE plus default-allow versus STRICT plus default-deny), egress (ALLOW_ANY versus REGISTRY_ONLY plus ServiceEntries), control plane revision (rollout skipped versus current revision) and limits (none or generous, hardly any load versus tightly set, under real load). The staging column carries a red deviation marker in every row, while the production column is the reference state. Below, the parity checklist: check these five points per environment, document deviations along with the reason. Punchline: the cheapest incident is the one that happens in staging.
The deltas I find most often sit in the mesh itself. Five of them, in the order in which I meet them on incident calls.
Injection. In staging, the sidecar often does not run at all or runs in an older version, because the last rollout was skipped there. If you test without a proxy, you test a different network.
Policies. AuthorizationPolicy and PeerAuthentication are frequently set more loosely in staging: PERMISSIVE instead of STRICT, default-allow instead of default-deny (report #15). A request that goes through in staging says nothing about whether it even arrives in production.
Egress. If staging has stayed on the old ALLOW_ANY (report #16) and production has long been on REGISTRY_ONLY, an external call reports success in staging and runs into a wall in production, because the ServiceEntry is missing.
Revision. Two environments on two different control plane revisions (report #20) are two different meshes with similar names.
Limits. In staging, the sidecar runs without tight resource limits and without real load. In production, the same proxy runs tightly constrained and right at its limits, under the load that makes the difference. So if you measure the sidecar costs from report #22 in staging, you get clean numbers for a different system.
Most incidents that were “only reproducible in production” turned out in the end to be a subset of this list, drawn up after the fact, once the unplanned night shift was over.
A copy of production down to the last byte is never realistic. What is realistic is a parity checklist for these five points, plus a deliberate decision: where staging deviates, the deviation is recorded in the document, along with the reason. Everything else is an incident waiting for its production date.
The math behind it is simple. The cheapest incident is the one that happens in staging. Every deviation that staging waves through moves on to production. There it costs the same, only with massive customer feedback.
Staging that does not share at least the mesh configuration of production does not test your application. It tests a different system that happens to carry the same name.
Which of the five deltas is most likely to hit you: injection, policies, egress, revision or limits?
