The Diagnostic Path Every Team Gets

Last week I described two teams. One guesses for three hours, one is done in twenty minutes. The difference isn’t talent. It’s this one path, and here it is.

Step 0: Read first, act later. Write down the exact error message and the Envoy flag before anyone restarts a pod. A restart that “gets it working again” erases the trail. Next time, you’re back at square one.

Step 1: istioctl proxy-status. Does the control plane even know the pod, and is its config SYNCED? If the pod doesn’t show up, the sidecar is missing and you’re looking at last week’s trivial WRONG_VERSION_NUMBER. If it says STALE, the proxy hasn’t pulled the new config. A distribution problem rather than a config error.

Step 2: istioctl analyze. The static check catches in seconds what you’d otherwise hunt for hours: a forgotten injection label, a VirtualService pointing at nothing, two rules overwriting each other.

Step 3: istioctl proxy-config. Now you look at what the sidecar actually sees, not what the YAML says. cluster and endpoints: are there any healthy addresses behind the target at all. route: does the route land where you think it does. secret: are the mTLS certificates rolled out.

Step 4: Back to the flag. Now it points at the hop. UF leads you to endpoints and mTLS. NR to the route. UO to the DestinationRule and its limits. The flag was half the diagnosis from the start. Now it’s all of it.

Step 5: Fix, then verify with the same path. proxy-status back to SYNCED, flag gone.

Five steps, always the same ones, in every incident. A team with this path no longer depends on the right person happening to be on call. Anyone can learn to apply the path. Knowing which five steps they are, and what the flag means in your specific mesh, is the experience behind it.

A runbook is unspectacular. That’s exactly why it holds up at three in the morning, when nobody is in the mood for heroics.

This is the runbook I build with teams in assessments, from the incidents they’ve already had. The method stays with the team. The experience that decides which steps those are is what I bring.

Which error message held you up the longest recently? Drop it in the comments and I’ll map it to its flag.