Skip to content

Engineering

LLM Observability With a Gateway: What Each Request Record Shows

What an LLM gateway records on every request: traces, sessions, the decision record, cost attribution by team, spend provenance and the efficiency score.

Nautili team

15 min read

TL;DR

  • Every keyed request through the Nautili gateway gets a decision record with the requested, decided, and served model; the serving provider; a switch reason; and any degradation, so an unintended model swap is visible in your data the day it happens.
  • Traces list every request with latency, cost, tokens, cache outcome, and status; Sessions group requests by session id, p95 latency shows per service and inside trials; and Evals test other models on live traffic without users seeing them.
  • Request headers tag cost to a user, team, project, and environment; X-Dz-Labels takes any key=value pairs, and five attribution surfaces (People, Teams, Departments, Services, and Seats) give a finance or platform lead views of who spent what.
  • Every dollar figure says who counted it and what kind of money it is, cost splits into uncached input, cache read, cache write, and output, and the limits are stated plainly: own-key spend is for visibility and not billing, and seat dollars are estimates at API rates.

Teams adopt LLM APIs one feature at a time, so six months later the answers to "who spent what" and "did a model change slip into a release" sit in several provider dashboards that do not agree.

This post covers what LLM observability looks like once every LLM call passes through a single gateway: traces, sessions and evals, the per-request decision record, the tags that attribute cost, how to read a spend figure, and the efficiency score. For how routing choices get made, read the routing profiles guide, and for budgets and access rules, see the governance guide.

The examples follow one question a platform lead might ask: why did last week's bill rise, and which team, service and request is behind it.

What is LLM Gateway Observability?

LLM observability is the practice of recording what happens on each call to a model, so you can answer four questions later: who sent it, which model answered, what it cost, and whether anything changed on the way. A normal application log holds some of that, but only for the code that wrote the log line, so every service has to be instrumented separately.

A gateway moves the recording to one place, so the record exists for every caller, whatever SDK or language they use. Nautili is an LLM gateway: you point an app or coding tool at it by changing a base URL and a key, and it routes each request, attributes cost, applies limits, and keeps the record.

Gateway observability rests on five parts:

  • Traces: every request with latency, cost, tokens, and status.
  • Sessions: requests grouped into conversations.
  • The decision record: which model was requested, decided, and served.
  • Attribution: who the spend belongs to.
  • Evals: whether a different model would do better.

LLM Monitoring vs LLM Observability, and Where a Gateway Fits

Monitoring watches known signals such as latency, error rate, and token counts. Observability keeps enough detail per request to explain a surprise you did not plan for.

Application tracing follows a call through your own code, and the OpenTelemetry project's GenAI conventions describe the attributes such traces carry, including the requested model and token counts.

A gateway record sits at a different boundary, capturing the request as it leaves your network, along with the caller's identity, the routing decision, and the price. The two complement each other, and the gateway record covers callers that never installed a tracing library.

Traces, Sessions and Evals: What the Dashboard Shows Beyond Cost

A record is only useful if you can find it, and the Usage and Evals screens turn records into something you can browse. Traces list every request with latency, cost, tokens, cache outcome, and status, and you can filter to errors, slow requests over 5 seconds, tool calls, or cached responses. The same page carries the prompt retention controls.

The Traces page in the Nautili dashboard, listing every request with trace ID, model, endpoint, status, user, project, cache outcome and latency, above filters for provider, model and cache.

  • The trace detail has a timeline split into gateway handling, provider time to first byte, response body, and gateway overhead.
  • Tabs show tokens, the prompt and response, tool calls, and attributes.
  • Traces carry OpenTelemetry gen_ai attributes and read a W3C traceparent header, so they line up with the conventions above.

A trace detail panel in the Nautili dashboard showing HTTP status, cache result, latency, cost and a timeline split into gateway handling, provider time to first byte, provider response body and gateway overhead.

Sessions group requests by session ID and show cost, tokens, duration, retries, and tool calls for each one.

The Sessions page in the Nautili dashboard, showing requests grouped by session ID with cost, tokens, duration, models used and retries.

Latency is reported as p95 in two places:

  • Per service, so the slowest 5% of a service's requests stay visible instead of disappearing into an average.
  • Inside trials, so a candidate model is compared with the current one on tail latency as well as quality.

Evals answer whether a different model would be better:

  • Live and Results show verdicts on live traffic or on your own corpus.
  • Trials shadow a candidate model on real traffic while users never see it.
  • Watches keep a replaced model as a sentinel after a promotion.
  • Compare runs one prompt across models side by side.
  • Datasets come from an imported file or are seeded from your traces.

Reading the Per-Request Decision Record Field by Field

On keyed traffic, the gateway writes a decision record for each request, including denials and cache hits, so that a model swap you did not intend shows up in your data before it shows up on your invoice.

FieldWhat it tells you
requested_modelWhat the caller asked for
decided_modelWhat the gateway resolved and gated on
served_modelWhat answered, parsed from the upstream's response
decided_upstreamWhich registered upstream connector served
served_providerWhich provider behind that upstream answered
routing_switch_reasonOne of three values: empty, upstream_fallback or auto_selection
routing_overrideWhich override fields the caller applied
routing_degradedWhat the gateway could not express to the upstream and proceeded without

requested_model is empty when the caller named no model, and served_model is empty on a denial or a cache hit. Compare requested_model with decided_model to see what routing resolved, then decided_model with served_model, the gap routing_switch_reason explains.

routing_switch_reason has three values:

  • Empty: no switch happened, and it is also empty when nothing was served.
  • upstream_fallback: the primary failed, and a model from the fallback chain served.
  • auto_selection: the served model is the pick of auto mode.

A variant name of the same model, such as a :floor suffix, is not counted as a switch. Degradation has a specific meaning: a ranking or fallback chain that cannot be expressed degrades and gets recorded, while an allow or deny list that cannot be enforced refuses the request.

Response Headers and Request Tags

Two response headers appear on keyed inference: X-Dz-Request-Id, the identifier you use to find the request in the dashboard, and X-Cache, which reports MISS, HIT, or SEMANTIC-HIT. On the request side, there are three groups of headers, all stripped before the request reaches the provider:

  • Attribution tags: X-Dz-User, X-Dz-Team, X-Dz-Project and X-Dz-Env, plus X-Dz-Labels, which takes any key=value pairs and stores them per request.
  • Source headers: X-Dz-Namespace, X-Dz-Workload and X-Dz-Pod.
  • Behavioral headers: X-Dz-Session-Id, X-Dz-Workload-Type and X-Dz-Router.

X-Dz-Session-Id is the only client-supplied identity that stays stable across turns (on the Anthropic Messages surface, Claude Code's own conversation ID also works). Cache-miss classification depends on it, which makes it the cheapest way to improve your own measurements. Here is a request from a batch pipeline:

curl -i https://<your-gateway-host>/v1/chat/completions \
-H "Authorization: Bearer sk-dz-<your-key>" \
-H "Content-Type: application/json" \
-H "X-Dz-User: <user>" \
-H "X-Dz-Team: data" \
-H "X-Dz-Project: churn-model" \
-H "X-Dz-Env: production" \
-H "X-Dz-Labels: <label>" \
-H "X-Dz-Session-Id: <session-id>" \
-d '{
"model": "anthropic/claude-sonnet-5.5",
"messages": [{"role": "user", "content": "Summarize this ticket."}]
}'

With the OpenAI SDK, the same tags go through default_headers, and with_raw_response exposes the two return headers. Model IDs use the author/slug form, so a native spelling like gpt-4o is rejected.

Five Attribution Surfaces for Tracing Spend to an Owner

Attribution turns a pile of token charges into a conversation between teams. Without it, the only split is by API key, and keys are shared more often than anyone admits, while a tag on the request travels with the call. When sources conflict, a rule configured by an administrator fills in a team for a known workload, the owner of an issued key overrides that rule, and a tag the caller sends explicitly wins over both.

Attribution has five surfaces that each answer a different question:

  • People: one human as telemetry knows them, including unlinked vendor accounts, which still appear as rows with spend and a team.
  • Teams: managed teams you curate with members and a monthly figure, or observed team-tag values with no managed record.
  • Departments: groupings of managed teams, or observed project-tag values.
  • Services: the backend installations calling the gateway, with 24-hour requests, spend, tokens, an efficiency label, error rate, status, and session count.
  • Seats: subscription seats observed sending traffic, which is what has been seen and never what you bought, so a purchased seat that never appears points to a dormant license.

The People page in the Nautili dashboard, attributing each request to a person with their team, department, request count, spend, sessions and status.

Tag values are free strings and case-sensitive, so checkout and Checkout become two projects, which is worth a naming convention before rollout.

X-Dz-Labels carries any key=value pairs you choose, such as a cost center or a feature name. Helicone's custom properties and Portkey's metadata also take arbitrary keys.

The Teams and Services views then show which team or backend service an increase came from.

A name sent only in X-Dz-User shows up on the People page prefixed unresolved::, for example unresolved::priya, until it is linked to a person on the Linking tab.

Managed team and department monthly figures enforce nothing, and only a guardrail budget in enforce mode refuses requests. The LLM governance guide covers budgets, rate ceilings, and the usage API fields that show spend against a limit.

Spend Provenance and Billing Regimes in the Spend Screen

The spend screen splits money into managed provider keys and Your own provider keys, and each arm names who counted it. Provider-attested means the figure comes from the upstream provider's own ledger, which applies to managed provider keys spend once the usage sweep has read the period. Gateway-reported means your gateway measured it as requests passed, and it is the only kind of figure that can exist for your own credentials.

Comparison of gateway-reported and provider-attested spend: who counted it, what it applies to, when it is available and what to use it for.

The billing regime is the second label: metered is real invoice dollars, subscription dollars are estimated at API rates and not charges, and unclassified has no resolvable credential.

The Spend page in the Nautili dashboard, with daily spend bars and a split between spend on managed provider keys and spend on your own provider keys.

For example, engineers on seat subscriptions see estimated dollars at API rates, which are not charges. A service on metered keys produces real dollars, so putting both on one chart without the labels would mix two kinds of money.

Cost Anatomy: The Four Parts of an LLM Bill

The spend page breaks cost into four parts: uncached input, cache read, cache write, and output. Uncached input is roughly 1 to 3% of an agentic Anthropic bill, while cache read dominates on agentic clients. Cache write exists for Anthropic only, so it shows as absent (never zero) for other providers.

The split follows how providers price caching. Anthropic's prompt caching documentation prices a five-minute cache write at 1.25 times the base input price and a cache read at 0.1 times, with some newer models lower. Nautili's view adds the attribution on top, so the four parts are available per provider, team, and service.

Bar showing where an agentic Anthropic bill goes: uncached input is roughly 1 to 3 percent, cache read dominates on agentic clients, cache write is Anthropic only, and the rest is output.

The Overview screen has a Where the money goes card that shows this split per provider, and it warns and suggests a shorter window when prices change during the range.

What the Efficiency Score Measures and How to Read It

The score is built from two pillars of waste (retry waste and prompt repetition waste) with the formula 100 multiplied by one minus wasted tokens over total tokens. Scores of 90 and above read as healthy, 75 to 89 as watch, and below 75 as high waste. The score is absolute, never relative to peers.

The score belongs to a person, and a team's score is a token-weighted average of its members. A person with only an attribution tag and no vendor identity has no score at all, which is different from a score of zero, and services carry a separate label instead of a score.

The where-to-look-first list ranks people by recoverable waste, a person's waste share multiplied by their volume, so a poor score on tiny volume ranks below a mediocre score on heavy use. For example, an engineer who sends a long system prompt on every turn would show repetition waste, and one whose pipeline retries on timeouts would show retry waste.

Checking Whether Each Source Is Live, Idle or No Data

A record is only as complete as the sources feeding it. The Sources screen shows each connected source as Live (a request in the past 24 hours), Idle (connected, no recent traffic), or No data (no request at all). Install verification is passive, so a source stays at No data until the first call arrives; every new team already has one source: the Shared Gateway.

The Sources page in the Nautili dashboard, listing the Shared Gateway every team has, marked Shared, Live and Operational.

This screen answers a common question before anyone blames the dashboard: did a new pipeline really point at the gateway? If its source shows No data, the traffic still goes straight to the provider, and none of the records above exist for it.

Limits You Should Know Before Relying on These Numbers

  • Own-key spend is for visibility, not billing, because the gateway counts what passed through while the provider invoices what it charged.
  • The Managed arm's daily bars are gateway-measured while its total uses the provider's monthly figure, so the bars may not sum to the total.
  • A seat is a flat monthly fee, so any dollar shown against a seat is an estimate of value at API rates and not a charge. Nothing on the seat surface is a cost-savings claim.
  • Telemetry is queued without blocking the response, which keeps the gateway fast and also means a row can arrive after the answer.
  • Telemetry observes and never denies a request. Enforcement lives in guardrails, covered in the governance guide.

Where to Start With Gateway Observability on Your Own Traffic

A gateway earns its place in observability by writing one record per request, with the requested, decided, and served model, the provider, the switch reason, and any degradation, tagged to an owner and priced with a stated source and regime. The numbers worth trusting carry their labels, and the limits are written down instead of left for you to find out later.

The practical first step is small. Point one service at the gateway using the quickstart, send the attribution headers from the example above, and read the traces and decision records for a day of traffic. Every team already has a Shared Gateway, so a first test needs no install, and Nautili is free to try.

Log retention depends on the plan, with a 7-day activity log on Free and 90-day logs on Pay-as-you-go.

Frequently Asked Questions

What is LLM observability, and what does a gateway add?

LLM observability is the record of who called which model, what it returned and what it cost. A gateway adds one place where that record is written for every caller, whatever SDK they use.

How do I tag LLM cost to a team or project?

Send the X-Dz-Team, X-Dz-Project and X-Dz-Env headers with each request, or set them once in your client. X-Dz-Labels adds any key=value pairs, and the gateway strips every X-Dz header before the request reaches the provider.

Why do my gateway totals and provider totals differ?

The gateway reports what it measured on your own provider keys, and your provider invoices what it charged, so the two can differ. For managed spend, the daily bars are gateway-measured while the total uses the provider's monthly figure.

Does a gateway replace application tracing?

No. Tracing explains what your code did before a call, and the gateway record explains the call itself: the caller, the decided and served model, the cache result, and the cost. Teams usually want both.

Written by

Nautili team

View all articles

Stay in the loop.

Product updates, technical guides and infrastructure thinking from Nautili.

No spam. Unsubscribe anytime.