Skip to content

Experiments, datasets and Compare: judge models before traffic

Import a dataset, price and run an experiment over it, and compare models side by side in Nautili, then know exactly what each result can and cannot decide.

13 min readLast reviewed October 19, 2026Verified against the Nautili dashboard and the LLMDatasetService and LLMTrialService APIs as of that date

What you’ll have at the end

By the end of this guide you will have a 30-row support-ticket triage dataset imported into Nautili, a quoted and completed experiment over it, and a Compare run with three models side by side. You will also know which of those results can change production routing, which is none of them, and how to graduate to the one that can.

Before you begin

  • A Nautili account with a team. Datasets, experiments and Compare are team-owned.
  • A provider credential connected under Provider keys. Experiment arms and Compare answers are dispatched on your team's credential.
  • A routing profile under Routing. An experiment requires a scope profile even though it never promotes into it.
  • The sample file: nautili-sample-dataset.jsonl.

Step 1: Understand traffic evidence versus corpus evidence

Nautili draws a hard line between two kinds of evidence, and the interface enforces it rather than suggesting it.

Produced byMeasured on your trafficTrials and replay trialsMeasured on a corpus you suppliedExperiments
Arms dispatchedMeasured on your trafficCandidate only; the incumbent already answered a userMeasured on a corpus you suppliedBoth, so about 2x cost per item
Priced before it runsMeasured on your trafficNoMeasured on a corpus you suppliedYes
Verdict wordingMeasured on your trafficAdoptable, Adopted, RejectedMeasured on a corpus you suppliedCandidate ahead, Parity, Candidate behind
Verdict may promoteMeasured on your trafficYes, manuallyMeasured on a corpus you suppliedNever

The load-bearing rule: corpus evidence can inform, only traffic evidence may promote. A corpus never touched the population a model is about to serve. The refusal lives in the API, not only in a disabled button: promoting from an experiment returns FailedPrecondition with this message.

an experiment's verdict speaks for its corpus, not for your traffic: promotion needs evidence from a trial on live or replayed requests

Results shows the two kinds in two tables that never interleave, headed Measured on your real traffic and Measured on a corpus you supplied. A corpus row never says "adoptable". Compare is a third thing, lighter than both: it stores nothing and never becomes evidence.

Step 2: Prepare a dataset file

A dataset is a stored, team-owned corpus. Each item has one required input, an optional expected output, and optional flat metadata used for filtering.

The importer accepts CSV and TSV with a header row, and JSONL with one JSON object per line (.csv, .tsv, .txt, .jsonl, .ndjson, .json). Braintrust and promptfoo exports work as they are. Roles are recognised by header name after normalising, so expected_output, expectedOutput and promptfoo's __expected all match. Recognised input headers include input, prompt, question, query, user_message, instruction, messages and text; recognised expected-output headers include expected_output, expected, golden, ground_truth, reference, answer, completion, response and output. In JSONL, metadata goes under a top-level metadata object.

The sample file uses input, expected_output and metadata. One row, trimmed to fit:

{"input":"Triage this support ticket. Reply with a JSON object with keys category, priority and first_response.\n\nTicket: I was charged twice for my monthly plan on the 3rd. ...","expected_output":"{\"category\":\"billing\",\"priority\":\"high\",\"first_response\":\"...\"}","metadata":{"category":"billing","priority":"high","channel":"email"}}

Its metadata.category (5 values), metadata.priority (4 values) and metadata.channel (3 values) keys are short and low-cardinality, so they become filterable dimensions. A top-level metadata object is flattened one level and the keys keep the metadata. prefix.

Limits are stated on the import screen under What we accept: 16 MB per file, 50,000 rows, 200 columns, 1,000,000 cells. Over any of these the upload fails. Per row, an input that is blank or longer than 256 KB skips that row and the skip is counted. An expected output over 256 KB is dropped from its row and a metadata value over 1 KB is shortened; the row still imports. A column becomes metadata only while its values are at most 64 characters and at most 20 distinct values; otherwise it is ignored as free text.

Step 3: Import the dataset

Open Datasets and choose Import dataset, which lands on /evals/datasets/import. The page is one screen with no wizard, and the flow is two calls.

  1. Under 1 · Source, choose the file and check Dataset name. The file is sent to AnalyzeDatasetFile, which reads it and stores nothing.

  2. Under 2 · What we read, confirm the sentence the importer states: which column it read as the input, which as an expected output, and which as metadata. AnalyzeDatasetFile returns a reason for every column (known name, longest text, low cardinality or free text); the sentence spells the reason out only for columns it ignores, for example "looked like free text, so we are ignoring it".

  3. If the reading is wrong, or the importer was not confident and opened the dropdowns on its own, click Change any of that and set each column to Input, Expected output, Metadata or Ignore. Exactly one column must be the input, and changing it re-reads the file because the usable and skipped counts depend on it.

  4. Click Import N items. The mapping on screen is sent back with the file, so ImportDataset honours what you approved even if detection later changes.

The dataset detail page for the sample dataset, with the category, priority and channel facets, the matching count and the Run experiment button

For the sample file the screen shows 30 usable, the toast reads Dataset imported, and the list row shows name, origin Imported file, items, expected outputs, skipped and the import time. The skipped count is stored on the dataset permanently.

Step 4: Quote and run an experiment

Open the dataset from the Datasets list, filter by its dimensions (for example metadata.priority = urgent), and click Run experiment. The New experiment form carries your filter and has three parts beside a sticky Quote panel that re-prices on every change.

SectionFields
The corpusItem count and the facet filters, editable here because narrowing the corpus changes the quote.
What to compareName (optional), Scope (routing profile) (required), the incumbent picker and the candidate cards. Model ids are author/slug. The incumbent may not also be a candidate.
Spend capHard spend cap, a slider up to $2,000. Consent, not advice: reaching it pauses the run and marks the verdict provisional. Unset means uncapped.

The quote comes from QuoteModelExperiment before anything dispatches. Figures below are illustrative:

Items 4 of 30
Dispatches 8 (4 × 2 arms)
anthropic/claude-haiku-4.5 · incumbent $0.01
openai/gpt-5.6-luna $0.02
Judge $0.01
Total $0.04
Cap $5.00
Charged to your openai (byok), anthropic (managed) credential. Nothing to install.
  • Items is the count after the filter, which is the number you are approving. Rows skipped on import are excluded.
  • Dispatches is items times arms. The incumbent is dispatched once per item and its answer is reused across every candidate.
  • Judge is one call per item per candidate, because the judge scores a pair.
  • A model with no resolvable rate shows rate unknown and the total becomes at least a figure, a floor rather than a price.
  • Charged to names the credential and custody (byok or managed) per provider. A missing credential shows up here, not mid-run.

Click Start experiment · $0.04 (the quoted total sits on the button; a floor shows as ≥ $0.04) or Save as draft. The toast reads Experiment started and you land on the run under /trials/<id>. Each corpus item is its own session, so the 20-session minimum applies to items. Exhausting the corpus completes the run. Experiments always run in DevZero's control plane, including on a content-free deployment, because a corpus is content you deliberately uploaded.

Step 5: Compare three models on one prompt

Open Compare. Under Setup, pick up to 6 models in Models, enter a System and User prompt, set Answer length (one of 256, 512, 1024, 2048, 4096 or 8192 tokens, default 1024) and Temperature (0 to 1, default 0), then click Run once. Paste one sample input as the user prompt. Results stream in: the resolved slate first, then each column's answer as it lands, then its score once the judge returns, then run totals.

Each column reports output, latency, cost, an absolute score (1 to 5 over quality, faithful, concise, instruct and clarity), a status of ok, no_credential or error, and a stop reason.

Who pays: the answers are dispatched on your team's credential; the judging runs on DevZero's judge credential. Nothing is stored, so Compare has no list, no history and no detail route. The limit of 20 runs per user per 5 minutes is a playground guard, not a billing control.

Verify

  • Datasets list: the row shows origin Imported file, 30 items, 30 expected outputs, and a dash under Skipped because no row was skipped.
  • Dataset detail: facet dropdowns for metadata.category, metadata.priority and metadata.channel, and a matching count that changes as you filter. Run experiment is disabled when the match is empty.
  • Results: the experiment appears only under Measured on a corpus you supplied, with a verdict of Candidate ahead, Parity or Candidate behind beside a judge version. It never offers promotion.
  • Compare: after a run, a banner reads "Like what <model> did here? A one-off proves nothing — a trial proves it on your real traffic" with a Start a trial from this button.

Step 6: Graduate to a trial once traffic exists

An experiment tells you a candidate handles the cases you chose. A trial tells you whether it can replace what you run, on the traffic it would serve. Once the scope profile carries real requests:

  1. Click Start a trial from this on Compare, or open New trial with the same incumbent, candidate and scope profile.
  2. Set sample_pct and a stop bound (max_sessions, max_duration_seconds or spend_cap). Only the candidate is dispatched; users keep getting the incumbent's answer.
  3. Wait for at least 20 sessions. The verdict is insufficient_data, degraded, parity or improved, with a tie band of 0.10.
  4. Promote manually from an Adoptable verdict. In the same dialog you can tick Watch this adoption to keep the replaced model running as a sentinel at a 2% session sample under a monthly budget you set.

Options

Metadata filter

Set it on the dataset detail page or the experiment form; it is an AND over exact key and value matches. The quote's Items line reads N of 30 when a filter is active. An empty match disables Run experiment rather than hiding it.

Spend cap

When the quoted total exceeds Hard spend cap, the panel warns: "This run costs more than the cap. It will pause when the cap is reached and its verdict will be provisional." The run still starts. Evaluation spend is never bounded by your guardrail budget, which only governs real traffic.

Troubleshooting

Symptom or messagethe file is N bytes, over the 16777216 byte limit, the file has more than 50000 rows, the file has N columns, over the 200 column limit, the file has more than 1000000 cells (N columns x M rows); for JSONL, the file has more than 200 distinct fieldsCauseWhole-file boundFixSplit or trim the file. Whole-file bounds fail the upload rather than skipping rows.
Symptom or messageChoose exactly one column as the input. or Only one column can be the input.CauseNo recognised input header, or two columns mapped as inputFixOpen Change any of that and set one column to Input; the other is demoted automatically.
Symptom or messageNo rows are usable with this mapping.CauseEvery row has a blank input under the chosen columnFixPick a different input column or fix the file.
Symptom or messageN rows will be skipped bannerCauseBlank inputs or inputs over 256 KBFixExpected; the approved count is the count that imports and the skipped count is stored on the dataset.
Symptom or messageno JSON objects were found in the fileCauseA .jsonl file that is not one object per lineFixMake each line a complete JSON object.
Symptom or messageQuote shows rate unknown and at least $XCauseA model with no contracted or list rateFixThe total is a floor. Set a cap you accept or remove the unpriced model.
Symptom or message"This run costs more than the cap"CauseQuoted total exceeds Hard spend capFixRaise the cap, narrow the filter, or accept a provisional verdict when it pauses.
Symptom or messageQuote credential error, or Compare status no_credentialCauseNo resolvable credential for that providerFixConnect one under Provider keys; the message names the provider.
Symptom or messageselect at most 50 traces at a timeCauseSeeding selection over the boundFixPromote in batches of 50.
Symptom or messagenone of the N selected traces could be promoted; call PreviewTraceSeeding for the reason on eachCauseEvery selected trace ineligible, commonly content_free_installation or outside_windowFixRun the preview, read each ineligible_reason, and select traces whose bodies were uplinked and are inside retention.
Symptom or messagean experiment's verdict speaks for its corpus, not for your trafficCausePromotion attempted from an experimentFixRun a trial on the scope profile; only a trial verdict can promote.

Next steps

Ready to keep going?

Explore the next guide or start routing through Nautili.