Observability That Pays for Itself
SLOs, tracing sampling, and alert design that reduce noise while cutting mean time to recovery on real incidents.
Cost without questions is waste
Observability spend balloons when every service logs at debug and traces at 100%. The question is not whether you have dashboards — it is whether those signals shorten incidents enough to justify the bill.
We tie instrumentation investment to user-facing SLOs. If a metric never informs a decision or alert, it is a candidate for deletion.
SLOs that drive behaviour
Pick a few latency and availability SLOs customers would recognise. Error budgets create permission to move fast and a forcing function to stop when reliability debt accumulates.
Alert on symptoms users feel, not on every disk spike. Page-worthy alerts should be rare, actionable, and tied to runbooks.
Tracing with intentional sampling
OpenTelemetry everywhere, sample intelligently. Keep head sampling high for rare error classes and critical paths; reduce volume on chatty success traffic.
Exemplars that link metrics to traces turn "p99 is high" into a specific slow dependency within minutes.
Noise reduction as an engineering goal
Track alert volume and acknowledge rates. If on-call ignores a class of alerts, fix the signal — do not hire people to endure it.
Observability that pays for itself shows up as lower MTTR and fewer repeat incidents, not as a prettier flame graph.
James Okonkwo
Platform Engineer
Platform engineer building event-driven systems, observability, and edge architectures.
