Skip to content

Engineering

LLM Governance in a Gateway: Budgets, Model Access and Content Rules

How LLM governance works inside a gateway: budgets and reset windows, model access lists, 20 deny reasons, content rules and a shadow-to-enforce rollout.

Nautili team

19 min read

TL;DR

  • A guardrail is a named bundle of budget, rate ceilings, model and provider access, and content rules, bound to a principal. Bundles never stack, so one guardrail governs a request, or none does.
  • Budgets are a USD hard ceiling with daily, weekly, monthly, or lifetime reset windows on a UTC calendar, counted per principal or pooled across a team on any guardrail. Requests-per-minute and tokens-per-minute ceilings sit beside them.
  • Every refusal maps to one of 20 fixed deny reasons, and the governance gate evaluates the model the router chose before any cache is consulted.
  • Content rules flag, redact, or block, and they scan requests only, a deliberate choice that keeps sensitive text from reaching a provider.
  • Roll out in shadow mode first, review every would-deny decision, then switch to enforce.

Teams rarely adopt LLMs all at once. A support team wires a bot to one provider, a data team starts a batch job against another, and a few engineers connect coding assistants to whatever key they were handed.

This post covers how LLM governance works inside a gateway, using Nautili's guardrails as the reference: budgets, rate ceilings, access lists, deny reasons, content rules, and the shadow-to-enforce rollout. Choosing which model answers a request is a separate topic, covered in our guide to routing profiles.

What Governance Means in an LLM Gateway?

An LLM gateway sits between your applications and the model providers. Because every call passes through one door, it can answer questions a single provider dashboard cannot: who is spending, on which models, how fast, and what text is leaving your network.

Governance is the set of rules the gateway applies at that door. In Nautili, a guardrail is a named bundle of those rules, and you attach it to a principal, which is the person, team, or service key the rule applies to.

Governance has two jobs:

  • Limiting money and capacity: how much a principal may spend and how quickly.
  • Limiting scope: which models, providers, and kinds of text a request may involve.

Those jobs map to four pillars: budgets, rate and token ceilings, model and provider access, and content rules.

How Gateway Governance Relates to AI Governance Frameworks

Search for LLM governance, and most results cover policy frameworks: who approves a model, how risk is documented, and which regulations apply. That work is necessary, but a written policy does nothing when a request is sent.

NIST's AI Risk Management Framework is voluntary guidance that organizes risk work into four functions: Govern, Map, Measure, and Manage. People own the policy, and a gateway makes it executable on every call: budgets and access lists apply the rules, while decision records and spend views supply the measurements.

To keep this concrete, suppose you run a software company with a support-bot team, a data team, and a coding-assistant group. Each has different risks, and each ends up with a different guardrail.

Where the Governance Gate Sits in the Request Path

Diagram of how the governance gate checks a request: the gateway resolves identity, model and billing regime, the governance gate checks the decided model before any cache, then rate, token and spend ceilings run before the request is forwarded to the provider. A refusal returns a deny reason.

The gateway reads the body and resolves the identity, the model, and the billing regime. The governance gate runs next, before the exact-match and semantic caches; then rate, token, and spend ceilings run, and only after that does the request go to the provider.

Two details carry most of the weight. The gate runs on the decided model, which is the model the router picked and not whatever string the client sent, so an allowlist stays meaningful when a routing profile swaps the model underneath the client. It also runs before any cache, so a cached answer cannot slip past a rule you wrote for a model.

How a Guardrail Binds to a Team, Identity or Service Key

Binding works as a short ladder: the gateway uses the principal's own binding if it has one, otherwise the team default, otherwise nothing. Nothing means allow, so a team with no configured guardrail is not governed at all.

Guardrails never stack, unlike OpenRouter's guardrails, where several guardrails apply at once, allowlists intersect, and the stricter rule wins. In Nautili, one guardrail governs a request, and routing profiles merge field by field. That keeps the answer to "why was this refused" to a single guardrail.

Budgets differ too. OpenRouter enforces guardrail budgets per user and per key rather than shared across everyone on a guardrail, while a Nautili guardrail can pool a budget per team with counter scope team.

Only ownerless service keys can be bound directly, and a person's key follows that person's identity guardrail. A new team already has a Default guardrail assigned as the team default, which runs requests on managed provider keys capped by the team's managed spend cap and has no other rules until you add them.

Diagram of how Nautili picks the guardrail for a request: the principal's own binding first, then the team default guardrail, and if neither exists the request is not governed. First match wins and guardrails never stack.

In this example, the support bot runs on an ownerless service key, so it carries its own guardrail with a tight model list. The data team gets a team-default guardrail covering every pipeline, and the coding-assistant group follows each person's identity guardrail.

Governance rules take effect within seconds. Content rules depend on the serving upstream: seconds where the gateway enforces, up to five minutes where an upstream does. Next, the first thing a bound guardrail limits: money.

The Guardrails page in the Nautili dashboard, listing the Default guardrail and a second guardrail with their status, budget, rules and assignments.

Budgets, Reset Windows and Per-Minute Ceilings Explained With Their Fields

The budget is a hard USD ceiling set from $0.0001 to $100,000,000, and unset or zero means no budget. The fields below make up the governance rules. Think of the budget as the outer limit and the rate ceilings as pacing inside it.

These controls address a risk that OWASP's 2026 Top 10 for LLM Applications lists as Unbounded Consumption (LLM06), where uncontrolled inference drives up cost and degrades service. OWASP's recommended mitigations include rate limiting, user quotas, and continuous monitoring of resource use; per-principal budgets with per-minute ceilings are the gateway form of the first two.

FieldWhat it doesValues
BudgetWhat it doesUSD hard ceiling for the windowValuesUnset or zero means no budget
Reset windowWhat it doesWhen the counter returns to zeroValuesDaily, weekly, monthly, or unset for lifetime; calendar UTC
Counter scopeWhat it doesWhose spend is counted togetherValuesPrincipal (default) or team, pooled across principals bound to the guardrail
Requests per minuteWhat it doesCeiling on request rateValues0 means unlimited
Tokens per minuteWhat it doesCeiling on token rateValues0 means unlimited
Soft-warn thresholdsWhat it doesEarly announcementsValuesFractions above 0 up to 1, announced once per window, never change a verdict

Counter scope has the most consequences. With principal scope, every principal gets its own counter, so one heavy user cannot drain a colleague's allowance. With team scope, all of them draw from one pooled counter, and any guardrail can pool this way.

In this example, the data team might use team scope with a monthly ceiling because its pipelines share one budget line, while the coding-assistant group suits principal scope so each engineer sees an individual ceiling.

  • A monthly budget restarts on the first of the month in UTC, and a lifetime budget never resets, which fits a fixed pilot allowance.
  • Rate and token ceilings are your own limits and do not reflect upstream throttling.
  • Soft-warn thresholds give an early signal but never change whether a request is allowed.

Model and Provider Access Lists and How Wildcards Match

Access control uses four lists: allowed models, ignored models, allowed providers and ignored providers. A star matches any run of characters, and an entry without a star matches byte for byte.

  • Exclusion beats permission, so a model on both an allow list and an ignore list is refused.
  • An empty permit list means unrestricted, but a non-empty provider allow list fails closed on providers the gateway does not recognize.
  • Wildcards expand into provider slugs, and if the catalog cannot be read while wildcards are present, the decision is refused.

A non-wildcard allowlist entry must match the gateway's outgoing traffic exactly; otherwise it can produce cached false denials, so test entries in shadow mode first.

Every entry in a fallback chain goes through the same guardrail check, and with enforce on, blocked entries are dropped before the request is forwarded, so for the support bot a fallback outside its allowlist never happens.

The 20 Deny Reasons Grouped by What Went Wrong

There are twenty fixed deny reasons. The grouping below is added to make the list easier to scan, and the names are the product's own. Find the group first, then the exact reason, and match any alerting on the reason names.

GroupDeny reasonsWhat it means
Access listsDeny reasonsmodel_not_allowed, provider_not_allowedWhat it meansThe model or provider is outside the lists
Guardrail stateDeny reasonspolicy_disabled, policy_expired, policy_unavailableWhat it meansThe guardrail is off, past its date, or could not be loaded
AttributionDeny reasonsunattributed_denied, tenant_unresolvedWhat it meansUnattributed traffic was denied by the team default, or the credential names no team
LimitsDeny reasonsbudget_exhausted, rate_limited, tpm_exceeded, budget_unavailable, model_unpriced, shared_gateway_ceilingWhat it meansA ceiling was hit, the budget could not be established, the model has no price while a budget is on, or the team went over the Shared Gateway per-minute ceiling
Credentials and custodyDeny reasonscredential_unavailable, custody_unsupported_upstream, key_invalid, token_invalid, management_keyWhat it meansThe key or the forwarding credential failed a check, including a management key used for inference
MeteringDeny reasonskey_policy_not_metered, user_policy_not_meteredWhat it meansThe bound policy is not metered for this key or user

Some refusals are collapsed into a single shape, so a caller who is probing the system learns nothing about which rule fired. Look up the real reason in your own records rather than inferring it from the client's response.

What a Refused Request Looks Like to the Client

With enforce on, a refusal uses each provider's native error shape, so the error handling already in your SDK keeps working:

  • OpenAI-compatible calls return an error object with message, type, code, and param, where code is the deny reason and param is the model.
  • Anthropic calls return {"type": "error", "error": {...}} with a type and a message.
  • Gemini calls return an error object with code, status, and message.

Status codes follow the kind of refusal:

  • Policy denials return 403.
  • rate_limited, tpm_exceeded, and shared_gateway_ceiling return 429 with a Retry-After header.
  • budget_exhausted returns 429 with insufficient_quota (400 on the Anthropic shape) and x-should-retry: false, so SDKs do not retry a spent budget.
  • A dead key returns 401.

Status codes vary across gateways. Portkey returns 446 when a guardrail check fails with deny on and 246 when deny is off, so a per-reason table like this one keeps client retry logic simple.

Content Rules: Flag, Redact or Block Before the Model Sees Text

Content rules scan requests and never model output. That is a deliberate choice: the point is to stop sensitive text before it reaches a provider, and a request is the only place that can happen.

LiteLLM's post_call mode and Portkey's output guardrails scan responses, and OpenRouter's guardrails describe request-side checks. If you need output checks on Nautili, run them in the application as well.

The built-in detectors cover email, phone, SSN, credit card, IP address, person names, postal addresses, and prompt injection. Prompt injection is the only detector that takes a scan scope, either all messages or user messages only, and each detector gets exactly one action.

Custom patterns use RE2 syntax, with at most 1,000 characters per pattern and 100 rules per list. Patterns compile on save, and a pattern that fails at request time refuses the request.

ActionWhat happensTypical use
FlagWhat happensThe match is recorded and the request proceeds unchangedTypical useFirst week on real traffic
RedactWhat happensThe match is replaced with a placeholder before the model sees itTypical useEmails and phone numbers in support tickets
BlockWhat happensThe request is refused before it reaches a modelTypical useCredit card numbers, card-like patterns

Flag, redact and block applied to the same request containing an email address: flag records it only, redact inserts a placeholder, and block refuses the request before the model.

Only detector categories are recorded, and the matched prompt text is never parsed, logged, or stored. Enforcement runs before the cache lookup, so two requests that differ only in a redacted value share a cache entry.

In this example, the support bot is where redact earns its place, since tickets carry customer emails and phone numbers, while the data team might block card numbers outright.

Setting Up These Controls in Nautili Using the Wizard, Keys and Headers

Guardrails are built in a five-step dashboard wizard (Identity and custody, Limits, Access, Content filters, and Review), where a seat guardrail omits the Limits step, and each step maps to a problem from earlier:

  • Identity and custody names the guardrail and picks whose credential it forwards with (Managed, BYOK, or Seat). A guardrail enforces nothing until something is assigned to it.
  • Limits holds the budget, counter scope, reset window, and per-minute ceilings, and Access holds the four lists.
  • Content filters hold the detectors and their actions, and Review is the last look before the guardrail goes live.

The guardrail wizard in the Nautili dashboard on the Identity and custody step, where a guardrail is set to Managed, BYOK or Seat credential custody.

Requests reach the gate through keys, which come in runtime and management kinds. Custody decides who pays the provider (managed, bring your own key, or auto), and an explicit binding that cannot resolve is denied and never falls back to the other party's credential. The X-Dz attribution headers tell the gateway who sent a request, and the LLM observability guide shows a full call with them.

Rolling Out From Shadow Mode to Enforce Without Breaking Production

The enforcement mode is set per gateway deployment: off, shadow, or enforce. Hosted deployments start in shadow, and self-hosted ones start off.

Shadow covers budgets, access lists, and rate limits for the whole gateway, and it records what each decision would have been. Log-only guardrails are not unique, since LiteLLM has a logging_only mode and Portkey's guardrails pass a failed check through when deny is false. What stands out is that shadow here also covers budgets and rate limits, which log-only guardrails do not.

One detail shapes the rollout. Requests that bring their own credentials, such as developer subscriptions and identity traffic, go through even when they would be denied. Requests on Nautili-issued keys still fail on a would-be-deny, so roll out shadow on credential-carrying traffic first and test key-based services in a staging team before they go live.

On a self-hosted gateway, issued keys need governance.issuedKeys and governance.byokForward on, and both depend on a governance mode:

governance:
issuedKeys: true
byokForward: true
mode: shadow # switch to enforce after the shadow review

A sensible rollout follows six steps:

  • Start the deployment in shadow and let normal traffic flow through it.
  • Make sure requests carry attribution, so would-deny decisions show whose they were.
  • Review every would-deny decision in the shadow window.
  • Fix allowlist entries and thresholds that are too narrow, then review again until nothing surprising remains.
  • Add content detectors as flags, then move the important ones to redact or block.
  • Only then ask for enforcement, or set the mode yourself on a self-hosted gateway.

Six-step rollout from shadow mode to enforce: start in shadow, attribute traffic with X-Dz headers, review every would-deny decision, fix allowlists and thresholds, add content detectors as flag then redact or block, and switch to enforce, repeating until there are no surprises.

In this example, the coding-assistant group is the likeliest source of surprises, because engineers try new models constantly, and shadow shows which experiments would have been refused before anyone is blocked.

Coverage Gaps Nautili States and How to Plan Around Them

Nautili is direct about limits, and a governance plan should include them:

  • Content rules do not cover requests sent through your own provider account; they are effectively team-wide, and identity, seat, and unattributed requests reach neither enforcer.
  • When an upstream enforces, a change can take up to five minutes to apply.
  • A team-default guardrail can govern unattributed requests normally, drop access rules for them, or deny them, so send the attribution headers to make every request belong to someone.
  • Managed team and department monthly figures enforce nothing, and only a guardrail budget in enforce mode refuses requests.

Reading Spend Against a Budget Without Misreading the Numbers

The usage endpoint of the management API returns the data you need, and each field has a fixed meaning:

FieldWhat it tells you
policy_idThe guardrail governing the principal, null if none
sourceWhose credential the guardrail forwards with: none, seat, byok, or managed
activeWhether the guardrail is in force now (enabled and not expired)
counter_scope, pooledprincipal or team; pooled means a shared team counter
limit_micro_usdThe hard budget cap, null if uncapped
window_start, resetStart of the current window and the next reset; reset is null for lifetime budgets
window_unboundedtrue for lifetime budgets
bucket_spent_micro_usd, bucket_as_ofThe counter spend enforcement uses and when it was last reconciled (about 10 minutes, best effort)
principal_spent_micro_usd, principal_spent_fromThis key's or person's own metered spend (successful requests only) and where the count starts
principal_spent_completeWhether that covers the whole window: always true for keys, can be false for people
remaining_micro_usdThe cap minus counter spend, never below 0; an estimate, since the live counter can be ahead
queried_atWhen the answer was computed

Seat guardrails return only the guardrail fields, with every money field null. The call needs a management-kind key with the usage:read scope, since a runtime key is refused with a 403. Amounts are in millionths of a dollar, so 640000 is $0.64, and a value that does not apply comes back as JSON null and never as zero.

Read the bucket figure to see how close a pooled team counter is to its ceiling, and the principal figure for what one person or service has spent. Check principal_spent_complete before relying on a principal total, and do not compare a budget with key spend shown elsewhere as an unfiltered cost over some number of days.

How to Start With LLM Gateway Governance in Nautili

Governance in an LLM gateway comes down to a few choices made in order: one guardrail per principal or team default, a budget whose counter scope and reset window match how your teams share money, rate and token ceilings, and access lists that are checked against the model the router decided. The gate runs before any cache and on the decided model, which keeps those rules honest.

You can try this on real traffic in a day, and every team already has a Shared Gateway, so there is nothing to install first:

  • Point one service at the gateway using the quickstart, and send X-Dz-Team, X-Dz-Project, and X-Dz-Env so every request has an owner.
  • Open Access > Guardrails, build a guardrail in the wizard with a budget and the model list that service needs, and assign it as the team default.
  • Leave the deployment in shadow and review every would-deny decision with the team that owns the traffic.
  • Add content detectors as flag first, then move the important ones to redact or block, and ask for enforcement once nothing surprising is left.

When the first month of data is in, trace individual decisions with the LLM observability guide. Try Nautili free to set up your first guardrail, and move to Pay-as-you-go when you want budgets.

Frequently Asked Questions

What is LLM governance, and how is it different from AI governance?

LLM governance is the set of rules applied to model calls at runtime: who may spend, on which models and providers, and what text a request may contain. AI governance is the wider organizational practice of policy, risk, and compliance, and gateway rules are one way to enforce it.

Can one guardrail stack with another guardrail on the same request?

No. A guardrail is a single bundle, and it governs a request or it doesn't. If you need different limits for different groups, create separate guardrails and bind each to the right principal.

Does a routing fallback chain bypass a guardrail's model allowlist?

No. Every chain entry goes through the same guardrail check, and with enforce on, blocked entries are dropped before the request is forwarded.

Do governance rules apply to coding assistants that use their own subscriptions?

Only partly. On a self-hosted or dedicated gateway, coding tools on their own subscriptions land on passthrough traffic, and guardrail enforcement needs an issued key, so issue keys for the coding-assistant group if you want budgets and access lists to apply. The Shared Gateway only accepts issued keys.

What happens when a request is refused?

The client receives an error in its provider's native shape, with the deny reason in the code field on OpenAI-compatible calls. Policy denials return 403, rate and token ceilings return 429 with Retry-After, and a spent budget returns 429 with x-should-retry: false.

Written by

Nautili team

View all articles

Stay in the loop.

Product updates, technical guides and infrastructure thinking from Nautili.

No spam. Unsubscribe anytime.