Observability that actually helps in an incident

During an incident, most people open Grafana first. Request rate, p99, CPU, all green. And that’s exactly where the team gets stuck.

Because green only means one thing here: on average, everything is fine. It says nothing about where, in a chain of five services, the one request this incident is actually about dies. The aggregate beats the single case, and an incident is always the single case.

The signal that actually helps is less glamorous. istio_requests_total, broken down by source and destination, by response_code and response_flags. One query, and “something is hanging” becomes “service A to service B, 503, flag UF”. The exact flag from the previous report, now visible across the entire mesh instead of in a single pod’s log.

With multiple hops, the trace comes into play. The span chain shows the hop where latency explodes or the call breaks off. Here’s the trap: Istio generates the spans, but propagating the trace headers is up to the services themselves. One service that swallows the header, and the chain breaks. Then you have gapless dashboards with a gap exactly where it hurts.

Kiali turns that into a picture. The graph shows the red edge where the traffic tips over. Three letters become a topology.

The point is not the tooling. Prometheus, Kiali, and tracing already come bundled with most Istio setups. What’s missing is the decision about which question your team must be able to answer when things go wrong, and making trace header propagation mandatory for every service instead of optional.

Which signals we really need in an emergency, and that every service propagates the trace headers, no exceptions: that’s the first thing I nail down in assessments. Not because the dashboard is missing. Because it answers the wrong question.

A green dashboard is not proof that things are working. It’s proof that nobody is measuring the right question.

Which signal actually pointed you to the dying hop in your last incident? And which green dashboard gave you the silent treatment?