Engineering
LLM Routing Profiles: Balance Cost and Quality
How LLM routing profiles balance cost and quality for each team: profile merging, auto mode and its cost tier, session pinning, fallbacks and rollout.

TL;DR
- An LLM routing profile is a named bundle of rules that tells a gateway which model a team's requests should use, which providers to prefer, and what to try when the first choice fails. In Nautili, it is scoped to a team and can be narrowed for an identity, an issued key, or a single request.
- Profiles merge field by field down a chain of team default, identity, key, and a per-request override, so a key can change one setting and inherit the rest. That keeps each team's cost and quality choices small and easy to audit.
- Auto mode lets the gateway choose a model per session based on a cost tier, and it requires an explicit opt-in on the profile.
- The first routing decision in a session is pinned and reused, and fallback chains of up to four models beyond the primary let a team plan for failures without letting a forbidden model slip in.
- Savings from routing are worth measuring with the gateway's own spend views and trials. This guide sticks to mechanics and leaves savings figures for your own traffic to show.
Choosing a model used to be a one-line decision in a config file. Once several teams share the same AI budget, that line carries real weight.
A product-description generator that writes thousands of short listings a day and a contract-review assistant that reads long agreements do not need the same model, and neither should be free to pick any model on the market. LLM routing settles this by matching each request to a model using rules the organization wrote in advance.
This guide walks through how routing profiles work. It covers:
- How a profile is built and how the cost tier works.
- Why a conversation stays on one model and how fallbacks behave.
- How auto mode and its cost tier work, including the setup in Nautili and a way to check that routing does what you intend.
What an LLM Router and a Routing Profile Are in Plain Terms
An LLM router is a service that sits between your application and the model providers. Your code sends one request to one endpoint, and the router decides which model and which provider answers it. A routing profile is the set of instructions the router reads when it makes that call.
In Nautili, a routing profile is a named bundle of routing behavior scoped to a team, and it holds:
- A default model.
- An ordered list of fallback models.
- Provider preferences.
- An auto mode switch with its cost tier, and a few more fields.
Think of it as a policy sheet that travels with the team. The listing team's sheet says which default model to use, what to fall back to, and which providers to prefer, and the contract-review team has its own.
Nobody edits application code to change a model; they edit the profile, and the change reaches traffic shortly after. The profile is the piece that ties routing choices to teams instead of to individual developers.
Why Each Team Needs Its Own Cost and Quality Balance
Say a company starts with a single model for everything, picked by whoever set up the first integration. It works until the invoice arrives, and then each team shows a different need:
- The listing generator sends many short, repetitive requests, where a cheaper model may be good enough.
- The contract-review assistant sends fewer, harder requests, and a wrong answer costs a lawyer's time.
A single global setting forces all three into one compromise. Per-team profiles let each group state its own preference, and an approved model list sets how much freedom each one gets.
The listing team can lean toward cost and the contract team toward quality. The gateway is where policy lives, so policy can differ per team without touching application code.
Common LLM Routing Strategies and What the Research Shows
LLM routing strategies fall into a few named families:
- Cost: send each request to the cheapest model or provider that qualifies.
- Latency: prefer the fastest provider.
- Semantic: pick a model from what the prompt is about.
- Cascade: try a cheap model first and escalate when the answer looks weak.
- Load balancing: split traffic across providers or keys.
- Fallback: move to a backup when the first choice fails.
The table shows how those families map onto Nautili. Canary releases and A/B splits send a slice of live traffic to a candidate model. Nautili has no direct equivalent, and the closest is a trial, where the candidate runs on live traffic while users never see it.
| Strategy | How it decides | Nautili equivalent |
|---|---|---|
| Cost or latency | How it decidesRanks providers by price, speed, or throughput | Nautili equivalentprovider_sort and provider order in the profile |
| Learned or auto | How it decidesPredicts the right model from the prompt | Nautili equivalentAuto mode with a cost tier |
| Fallback | How it decidesMoves to a backup on failure | Nautili equivalentFallback chain, up to four beyond the primary |
| Canary or A/B | How it decidesSends a slice of traffic to a candidate | Nautili equivalentClosest equivalent is a trial, where users never see the candidate |
Every router adds a hop, so latency overhead is worth measuring on your own traffic.
The best-known research is RouteLLM, an open-source framework whose published results report cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against using only GPT-4, while keeping 95% of GPT-4's performance. The spread is the lesson, because the same method saved very different amounts depending on the questions, and the authors note that the routers did poorly on MMLU when trained only on Arena data.
LiteLLM's own auto routing benchmark points the same way: its rule-based complexity router did not beat a cost-matched shuffled control. Treat any headline routing the same way. Your result depends on how much of your traffic is easy, so measure on your own requests, which is what the trials and spend views later in this guide are for.
How Routing Stays Inside an Approved Model List

Freedom to choose a model needs a boundary. If every team could route to any model, a cost tier would be a suggestion, so the boundary is an allowlist of the models and providers the organization has approved.
In Nautili, that boundary comes from guardrails, which are separate from profiles. Routing picks the model, and routing does not decide whether a request may spend, since guardrails own that decision. The governance rules follow a clear order of precedence: exclusion beats permission, and an empty allow list means unrestricted.
The order of operations is what makes the pairing work. Routing runs first and produces a decided model, then the governance gate checks the decided model rather than the string the client sent, before any cache is consulted.
That detail closes a loophole, because a client cannot sneak past the allowlist by naming a harmless model when the gate looks at what the router chose. Our guide to LLM governance covers wider policy design.
Why a Session Should Stay on the Model It Started With
Chat and agent workloads send many turns that build on earlier messages. A conversation that hops between models can change tone or behavior mid-task, so a router should decide once per session and reuse that decision. The gateway needs a way to tell sessions apart, and in Nautili that identity is the X-Dz-Session-Id header, the only client-supplied identity that stays stable across turns (on the Anthropic Messages surface, Claude Code's own conversation ID also works).
For the listing generator, one batch job is one session, and for the contract-review assistant, one document conversation is. When the application supplies a stable ID, the gateway can pin the first decision and reuse it on every later turn, and the pin only moves when a failure forces a fallback. Teams that skip the header get less reliable pinning, so it is one of the first things worth wiring up.
Pinning also protects cost. Anthropic's prompt caching documentation prices a cache read at 0.1 times the base input price by default, with a default cache lifetime of five minutes that refreshes on every use, so a conversation that keeps reusing its prefix pays less per turn.
How Fallbacks and Provider Preferences Behave Under Failure

A fallback chain is an ordered list of backup models, tried when the primary cannot serve the request. In Nautili, a chain holds up to four models beyond the primary, and the upstream runs the failover while the gateway records what served the request.
A chain is a plan for a failed attempt and is separate from retries. Every entry in the chain has to pass the same governance gate as the primary, so a forbidden model in the list is dropped rather than used.
There is also a useful distinction between preferences and restrictions:
- Preferences degrade: if a provider ranking cannot be expressed on a given path, the request proceeds and the loss is recorded.
- Restrictions refuse: an allow list, a deny list, or an auto mode that cannot be expressed stops the request.
The rule of thumb is to put what you would like in rankings and preferred order, and what you require in allow and deny lists. The next section shows how Nautili turns these ideas into a concrete setup.
How Nautili Builds Routing Profiles from Team to Request
A profile in Nautili is attached at one of four rungs, from least to most specific:
- The team default.
- An identity.
- An issued key.
- A per-request override.
Fields merge from rung to rung, so the most specific rung that sets a field wins for that field alone. Guardrails work differently, because they select one whole guardrail and never stack.
The practical result is that the listing team can hold one team default while a single service key tightens only the provider list. Profiles attach to issued keys, which are the keys Nautili itself creates, so a key given to one service carries that service's routing with it.

The fields on a profile cover the whole routing decision from default model to upstream. They are set in the dashboard wizard rather than in a config file, so the table below describes each field, what it controls, and one detail worth remembering when you edit it, since a few fields behave differently from what many teams first expect.
| Field | What it controls | Useful detail |
|---|---|---|
| Default model | What it controlsThe model used when a request names none | Useful detailWritten as author/slug, such as anthropic/claude-sonnet-5.5 |
| Fallback models | What it controlsOrdered backups when the first choice fails | Useful detailUp to four entries beyond the primary |
| Rank providers by | What it controlsCost, latency, or throughput ordering | Useful detailA preference, so it degrades |
| Preferred, allowed, denied providers | What it controlsWhich providers are favored or excluded | Useful detailA provider cannot be both allowed and denied |
| Auto enabled | What it controlsWhether model "auto" may be used | Useful detailYes, no, or inherit |
| Auto cost tier | What it controlsHow auto weighs cost | Useful detailDial of 0 to 10 that maps to a tier, default 5 (high) |
| Upstream | What it controlsThe connector that carries the traffic | Useful detailSets egress, not who pays |
Two details save confusion later: an empty list means "inherit", not "none", and updates replace the whole profile instead of patching it, so omitted fields are blanked. Keep both in mind whenever a team edits a profile.

Once saved, the profile appears in the Routing profiles list with its bindings, default model, fallback count, and auto setting on one row.

How the Cost Tier Works Inside Auto Mode
The third wizard zone, covering providers and auto, holds the cost tier control. Auto mode lets the gateway choose a model on a team's behalf, so a client sends model "auto" instead of a named model, and the choice is delegated to the serving upstream's auto-router.
The cost tier has five levels: low, medium, high, xhigh, and max, and the profile sets it with a dial from 0 to 10, where 0 favors quality and 10 favors cost, and each setting lands in one of the five tier bands below.
| Dial | Tier |
|---|---|
| 0 to 1 | max |
| 2 to 3 | xhigh |
| 4 to 6 | high |
| 7 to 8 | medium |
| 9 to 10 | low |
The default is 5, which is high. A team that sets 0 is asking for quality first.
OpenRouter's auto router uses the same five tiers, and with no setting it routes roughly as low, so Nautili's default is a higher band. Its auto router also works differently from a quality score: it classifies the prompt into a task type and ranks models by their share of spend for that task.
In this example:
- The listing team might set the dial toward cost.
- The contract-review team might set it toward quality.
- A team that wants one specific model might skip auto and name it.

Auto mode asks for explicit consent. If a profile has not enabled it, a request for model "auto" fails with an auto_not_enabled error, which Nautili sums up as "Configuration is consent".
Auto stays opt-in per profile, and guardrails still decide whether the request may spend.
One cost detail is worth knowing. Because the first pick is pinned per session, auto costs one uncached turn per session, and it needs a session ID to pin at all, so send the header from the first request.
Nautili forwards the session ID to the serving upstream as session_id. OpenRouter also keeps a conversation on its model, but only while that model stays a top candidate for each new prompt, so Nautili's pin and the upstream's memory are two separate layers.
A request that names a model skips auto, and the wizard states that nothing is ever substituted for a model a request named explicitly. A short example of an auto request, using a placeholder host and key, looks like this:
How a Single Request Can Steer Routing Without Editing the Profile
The top rung of the ladder is the per-request override. It rides in the X-Dz-Router header as compact JSON up to 8192 bytes, and it can change:
- Provider sorting and provider order.
- Allow and deny lists for providers.
- The fallback chain, the auto dials, and the upstream.
There is no model field, so an override cannot name a model, and a field the router does not know returns an invalid_dz_router error. An override can steer inside a mode the team enabled but cannot enable one, so asking for something the profile did not turn on returns routing_override_not_permitted:
- Auto dials on a profile without auto enabled.
fallback_modelson a profile with no chain and no auto.- A
provider.onlylist that names none of the profile's allowed providers.
An override can never widen the approved list, and nothing replaces a model a request names explicitly. The example below sorts providers by latency and denies one provider for a single call.
What Changes for Coding Tools Such as Cursor and Claude Code
One group needs extra care: engineers using coding tools. Coding tools often authenticate with the developer's own subscription or provider key. On a self-hosted or dedicated gateway that traffic passes through: it gets telemetry and per-developer attribution but not routing profiles, which need an issued key. The Shared Gateway only accepts Nautili-issued keys, so there coding tools use an issued key from the start.
The LLM observability guide covers what that traffic still records, and the team decides whether profile control is worth changing how its developers sign in.
How to Check Whether Routing Is Saving Money
Routing saves money only if you measure it against a fair baseline: compare like with like at the team level, and treat per-person numbers as estimates. Any figure you read about a specific routing saving, including from vendor pages, should be tested against your own traffic.
For quality, a trial asks whether a cheaper model would hold up. A trial dispatches only the candidate model on live traffic, so users are never exposed to a candidate they did not already get. A cheaper model that passes a trial is a good candidate for a profile's default.

Every keyed request also records the requested, decided and served model, along with the reason for any switch. Our LLM observability guide covers reading records like these.
A Practical Rollout Order for Teams With Different Needs
Start with the team that has the clearest, lowest-risk traffic, which in this example is the listing generator. Create the profile in the wizard, bind it as the team default, set a default model and one or two fallbacks, and wire up the session ID header. Leave auto off until the basics are stable.
Next comes the contract-review team, with a wider set of models and a cost tier that leans toward quality. Use trials to test cheaper models, and add auto once the team has tested it on a small slice of traffic.
Finally, decide how engineers' coding tools connect. Change one thing at a time so a shift in spend has one obvious cause.
Where Should You Start with Routing Profiles
A routing profile gives each team its own balance of cost and quality while the organization keeps one approved list. Profiles merge from team default down to a single request, the cost tier shapes what auto mode may choose, the first decision in a session stays pinned, and fallbacks and provider preferences degrade gracefully while allow and deny lists refuse when they cannot be met. The pieces fit because routing chooses the model and guardrails decide whether the request can proceed.
To start, pick the team with the simplest traffic, write its profile with an explicit model list, and send a session ID with every request. Then compare its spend before and after using the gateway's own views. You can try the gateway free and set up your first profile in the dashboard wizard.
Frequently Asked Questions
What is an LLM router?
An LLM router is a service between your application and the model providers that decides which model answers each request. It can follow fixed rules, a cost tier, or a learned prediction; in Nautili, routing profiles define the rules.
What is an LLM routing profile?
An LLM routing profile is a named bundle of routing settings that tells the gateway which model, providers, and fallbacks to use. In Nautili, it is scoped to a team and merged field by field, so an identity, a key, or a single request can change one setting and inherit the rest. Spending limits and model access belong to guardrails.
How does auto mode choose a model?
A client sends "auto" in the model field of the chat completions call instead of a named model. The pick is delegated to the serving upstream's auto-router, guided by the profile's cost tier. The first choice is pinned for the session.
Does routing stay inside the model allowlist for named models?
For requests that name a model, yes. Routing produces a decided model, and the governance gate then checks that model against the allowlist before any cache or forwarding.
Why pin a model for the whole session?
Pinning keeps a conversation on one model, avoiding behavior shifts mid-task and protecting the cached conversation prefix. The gateway relies on a session ID, and the supported surfaces differ in how they supply it: the Anthropic Messages surface can use the client's conversation ID, while OpenAI and Gemini surfaces use only the X-Dz-Session-Id header. The pin changes only when a failure forces a fallback.
How do I measure whether routing saved money?
Compare spend for the same team before and after the change, grouped by model, and read through the gateway's spend views. Use trials to test whether a cheaper model keeps quality, and read the decision records to confirm which model served each request. Team-level comparisons are the reliable view.
Written by
Nautili team


