guided tour
Seven steps through one real incident
The workspace below is Northwind Commerce at 14:41 on a Tuesday. A deploy went out 31 minutes ago, the p99 on one checkout endpoint has gone from 218ms to 1.4 seconds, and three incidents are open. Step through the console the way the on-call engineer would.
Every service, one column
Services are listed with the three numbers that decide whether you need to care: requests per minute, p99, and error rate. Ownership comes from your CODEOWNERS file, so the rotation this pages is always the team that actually ships the code.
checkout-api is the only service not healthy — 1,412ms p99, 3.10% errors.
Step 1 of 7: Every service, one column. Services are listed with the three numbers that decide whether you need to care: requests per minute, p99, and error rate. Ownership comes from your CODEOWNERS file, so the rotation this pages is always the team that actually ships the code.
services
5- checkout-api1,412ms
- catalog-api184ms
- identity96ms
- fulfilment308ms
- webhooks-out742ms
owner: @northwind/payments
checkout-api · latency
p99 1,412msp95 966msp50 176ms5xx rate · peak 3.34%
deploys in window
2- 9d3be71checkout-api · 14:10suspected cause
- a41f0c2catalog-api · 12:00
slowest routes
by p99| route | p99 | err |
|---|---|---|
| POST/v2/checkout/:cartId | 1,412ms | 3.10% |
| POST/v2/checkout/:cartId/pay | 884ms | 1.40% |
| GET/v2/cart/:cartId | 212ms | 0.02% |
| GET/v2/products/:sku | 141ms | 0.01% |
| POST/v2/sessions | 96ms | 0.11% |
open incidents
3- SEV2INC-2291 · 14:32unacked
p99 latency above 1s on POST /v2/checkout/:cartId
@northwind/payments · Priya on call
- SEV3INC-2290 · 14:33ack 14:35
5xx rate above 1% on POST /v2/checkout/:cartId
@northwind/payments · Priya on call
- SEV3INC-2287 · 14:36ack 14:38
Novel log pattern: PoolTimeoutError, 312/hr
@northwind/platform · Sam on call
attached logs · INC-2291
novel pattern- 14:32:04warnpool.acquire waited 2140ms (size=10, in_use=10)
- 14:32:06errorPoolTimeoutError: could not acquire connection in 3000ms
- 14:32:06infocheckout.cart_id=c_8812f retry=1 backoff=250ms
- 14:32:09errorPoolTimeoutError: could not acquire connection in 3000ms
- 14:32:11warnpool.acquire waited 2884ms (size=10, in_use=10)
PoolTimeoutError · 312/hr · first seen today
service map
observedSample workspace. Static data, real layout — this is the console at 14:41 on the day of the incident.
what you just saw
The same seven panels, in order
Every one of them is on the default dashboard. There is no view to build and no query language to learn before the first incident.
01 · services
Every service, one column
Services are listed with the three numbers that decide whether you need to care: requests per minute, p99, and error rate. Ownership comes from your CODEOWNERS file, so the rotation this pages is always the team that actually ships the code.
02 · latency
Percentiles per route, not per service
The chart is p50, p95, and p99 for the selected service across the last six hours. Averages hide this shape completely: p50 barely moved, which is why a service-wide mean would still look almost fine while one in a hundred customers waits a second and a half.
03 · deploys
Deploys drawn on the same axis
Deploy markers land on every chart, pulled from your CI. Tracepoint ranks the ones touching code that owns the affected route first, so the first question of any incident — did we ship something — is answered before anybody has to ask it in Slack.
04 · routes
Down to the specific endpoint
Routes are bucketed by pattern, not by URL, so /v2/checkout/:cartId stays one series instead of fragmenting into a series per cart. Sorting by p99 puts the endpoint that broke at the top, with the request volume next to it so you can tell a real problem from a rare one.
05 · incidents
Incidents, already routed
Three signals moved and three incidents opened, each routed to the on-call engineer for the owning team. Severity is derived from the rule that fired and how much traffic it affects; acknowledgement and escalation happen in Slack without anyone opening this page.
06 · logs
The log lines, already attached
Tracepoint fingerprints log lines into patterns and learns each pattern's normal volume. PoolTimeoutError had never been seen in this workspace before today and appeared 312 times in an hour, so it raised itself and attached to the open incident rather than waiting for someone to think of grepping for it.
07 · map
Where it stops being your problem
The service map draws dependencies from observed traffic, with the hot path highlighted. Here the degradation propagates from checkout-api to the payments database and no further, which rules out the upstream edge and identity in about one second of looking.
The console is the easy part. The routing is why it works.
Finding the regression is one thing; getting it to the engineer who can fix it, at 2am, without waking four others, is the part teams usually never finish building. Alert rules live in your repo and resolve owners from CODEOWNERS.