Experiments, datasets and Compare: judge models before traffic
Import a dataset, price and run an experiment over it, and compare models side by side in Nautili, then know exactly what each result can and cannot decide.
13 min readLast reviewed October 19, 2026Verified against the Nautili dashboard and the LLMDatasetService and LLMTrialService APIs as of that date
What you’ll have at the end
By the end of this guide you will have a 30-row support-ticket triage dataset imported into Nautili, a quoted and completed experiment over it, and a Compare run with three models side by side. You will also know which of those results can change production routing, which is none of them, and how to graduate to the one that can.
Before you begin
- A Nautili account with a team. Datasets, experiments and Compare are team-owned.
- A provider credential connected under Provider keys. Experiment arms and Compare answers are dispatched on your team's credential.
- A routing profile under Routing. An experiment requires a scope profile even though it never promotes into it.
- The sample file: nautili-sample-dataset.jsonl.
Step 1: Understand traffic evidence versus corpus evidence
Nautili draws a hard line between two kinds of evidence, and the interface enforces it rather than suggesting it.
| Measured on your traffic | Measured on a corpus you supplied | |
|---|---|---|
| Produced by | Measured on your trafficTrials and replay trials | Measured on a corpus you suppliedExperiments |
| Arms dispatched | Measured on your trafficCandidate only; the incumbent already answered a user | Measured on a corpus you suppliedBoth, so about 2x cost per item |
| Priced before it runs | Measured on your trafficNo | Measured on a corpus you suppliedYes |
| Verdict wording | Measured on your trafficAdoptable, Adopted, Rejected | Measured on a corpus you suppliedCandidate ahead, Parity, Candidate behind |
| Verdict may promote | Measured on your trafficYes, manually | Measured on a corpus you suppliedNever |
The load-bearing rule: corpus evidence can inform, only traffic evidence may promote. A corpus never touched the population a model is about to serve. The refusal lives in the API, not only in a disabled button: promoting from an experiment returns FailedPrecondition with this message.
Results shows the two kinds in two tables that never interleave, headed Measured on your real traffic and Measured on a corpus you supplied. A corpus row never says "adoptable". Compare is a third thing, lighter than both: it stores nothing and never becomes evidence.
Step 2: Prepare a dataset file
A dataset is a stored, team-owned corpus. Each item has one required input, an optional expected output, and optional flat metadata used for filtering.
The importer accepts CSV and TSV with a header row, and JSONL with one JSON object per line (.csv, .tsv, .txt, .jsonl, .ndjson, .json). Braintrust and promptfoo exports work as they are. Roles are recognised by header name after normalising, so expected_output, expectedOutput and promptfoo's __expected all match. Recognised input headers include input, prompt, question, query, user_message, instruction, messages and text; recognised expected-output headers include expected_output, expected, golden, ground_truth, reference, answer, completion, response and output. In JSONL, metadata goes under a top-level metadata object.
The sample file uses input, expected_output and metadata. One row, trimmed to fit:
Its metadata.category (5 values), metadata.priority (4 values) and metadata.channel (3 values) keys are short and low-cardinality, so they become filterable dimensions. A top-level metadata object is flattened one level and the keys keep the metadata. prefix.
Limits are stated on the import screen under What we accept: 16 MB per file, 50,000 rows, 200 columns, 1,000,000 cells. Over any of these the upload fails. Per row, an input that is blank or longer than 256 KB skips that row and the skip is counted. An expected output over 256 KB is dropped from its row and a metadata value over 1 KB is shortened; the row still imports. A column becomes metadata only while its values are at most 64 characters and at most 20 distinct values; otherwise it is ignored as free text.
Step 3: Import the dataset
Open Datasets and choose Import dataset, which lands on /evals/datasets/import. The page is one screen with no wizard, and the flow is two calls.
Under 1 · Source, choose the file and check Dataset name. The file is sent to
AnalyzeDatasetFile, which reads it and stores nothing.Under 2 · What we read, confirm the sentence the importer states: which column it read as the input, which as an expected output, and which as metadata.
AnalyzeDatasetFilereturns a reason for every column (known name, longest text, low cardinality or free text); the sentence spells the reason out only for columns it ignores, for example "looked like free text, so we are ignoring it".If the reading is wrong, or the importer was not confident and opened the dropdowns on its own, click Change any of that and set each column to Input, Expected output, Metadata or Ignore. Exactly one column must be the input, and changing it re-reads the file because the usable and skipped counts depend on it.
Click Import N items. The mapping on screen is sent back with the file, so
ImportDatasethonours what you approved even if detection later changes.

For the sample file the screen shows 30 usable, the toast reads Dataset imported, and the list row shows name, origin Imported file, items, expected outputs, skipped and the import time. The skipped count is stored on the dataset permanently.
Step 4: Quote and run an experiment
Open the dataset from the Datasets list, filter by its dimensions (for example metadata.priority = urgent), and click Run experiment. The New experiment form carries your filter and has three parts beside a sticky Quote panel that re-prices on every change.
| Section | Fields |
|---|---|
| The corpus | Item count and the facet filters, editable here because narrowing the corpus changes the quote. |
| What to compare | Name (optional), Scope (routing profile) (required), the incumbent picker and the candidate cards. Model ids are author/slug. The incumbent may not also be a candidate. |
| Spend cap | Hard spend cap, a slider up to $2,000. Consent, not advice: reaching it pauses the run and marks the verdict provisional. Unset means uncapped. |
The quote comes from QuoteModelExperiment before anything dispatches. Figures below are illustrative:
- Items is the count after the filter, which is the number you are approving. Rows skipped on import are excluded.
- Dispatches is items times arms. The incumbent is dispatched once per item and its answer is reused across every candidate.
- Judge is one call per item per candidate, because the judge scores a pair.
- A model with no resolvable rate shows rate unknown and the total becomes at least a figure, a floor rather than a price.
- Charged to names the credential and custody (
byokormanaged) per provider. A missing credential shows up here, not mid-run.
Click Start experiment · $0.04 (the quoted total sits on the button; a floor shows as ≥ $0.04) or Save as draft. The toast reads Experiment started and you land on the run under /trials/<id>. Each corpus item is its own session, so the 20-session minimum applies to items. Exhausting the corpus completes the run. Experiments always run in DevZero's control plane, including on a content-free deployment, because a corpus is content you deliberately uploaded.
Step 5: Compare three models on one prompt
Open Compare. Under Setup, pick up to 6 models in Models, enter a System and User prompt, set Answer length (one of 256, 512, 1024, 2048, 4096 or 8192 tokens, default 1024) and Temperature (0 to 1, default 0), then click Run once. Paste one sample input as the user prompt. Results stream in: the resolved slate first, then each column's answer as it lands, then its score once the judge returns, then run totals.
Each column reports output, latency, cost, an absolute score (1 to 5 over quality, faithful, concise, instruct and clarity), a status of ok, no_credential or error, and a stop reason.
Who pays: the answers are dispatched on your team's credential; the judging runs on DevZero's judge credential. Nothing is stored, so Compare has no list, no history and no detail route. The limit of 20 runs per user per 5 minutes is a playground guard, not a billing control.
Verify
- Datasets list: the row shows origin Imported file, 30 items, 30 expected outputs, and a dash under Skipped because no row was skipped.
- Dataset detail: facet dropdowns for
metadata.category,metadata.priorityandmetadata.channel, and a matching count that changes as you filter. Run experiment is disabled when the match is empty. - Results: the experiment appears only under Measured on a corpus you supplied, with a verdict of Candidate ahead, Parity or Candidate behind beside a judge version. It never offers promotion.
- Compare: after a run, a banner reads "Like what
<model>did here? A one-off proves nothing — a trial proves it on your real traffic" with a Start a trial from this button.
Step 6: Graduate to a trial once traffic exists
An experiment tells you a candidate handles the cases you chose. A trial tells you whether it can replace what you run, on the traffic it would serve. Once the scope profile carries real requests:
- Click Start a trial from this on Compare, or open New trial with the same incumbent, candidate and scope profile.
- Set
sample_pctand a stop bound (max_sessions,max_duration_secondsorspend_cap). Only the candidate is dispatched; users keep getting the incumbent's answer. - Wait for at least 20 sessions. The verdict is
insufficient_data,degraded,parityorimproved, with a tie band of 0.10. - Promote manually from an Adoptable verdict. In the same dialog you can tick Watch this adoption to keep the replaced model running as a sentinel at a 2% session sample under a monthly budget you set.
Options
Metadata filter
Set it on the dataset detail page or the experiment form; it is an AND over exact key and value matches. The quote's Items line reads N of 30 when a filter is active. An empty match disables Run experiment rather than hiding it.
Spend cap
When the quoted total exceeds Hard spend cap, the panel warns: "This run costs more than the cap. It will pause when the cap is reached and its verdict will be provisional." The run still starts. Evaluation spend is never bounded by your guardrail budget, which only governs real traffic.
Troubleshooting
| Symptom or message | Cause | Fix |
|---|---|---|
Symptom or messagethe file is N bytes, over the 16777216 byte limit, the file has more than 50000 rows, the file has N columns, over the 200 column limit, the file has more than 1000000 cells (N columns x M rows); for JSONL, the file has more than 200 distinct fields | CauseWhole-file bound | FixSplit or trim the file. Whole-file bounds fail the upload rather than skipping rows. |
Symptom or messageChoose exactly one column as the input. or Only one column can be the input. | CauseNo recognised input header, or two columns mapped as input | FixOpen Change any of that and set one column to Input; the other is demoted automatically. |
Symptom or messageNo rows are usable with this mapping. | CauseEvery row has a blank input under the chosen column | FixPick a different input column or fix the file. |
Symptom or messageN rows will be skipped banner | CauseBlank inputs or inputs over 256 KB | FixExpected; the approved count is the count that imports and the skipped count is stored on the dataset. |
Symptom or messageno JSON objects were found in the file | CauseA .jsonl file that is not one object per line | FixMake each line a complete JSON object. |
| Symptom or messageQuote shows rate unknown and at least $X | CauseA model with no contracted or list rate | FixThe total is a floor. Set a cap you accept or remove the unpriced model. |
| Symptom or message"This run costs more than the cap" | CauseQuoted total exceeds Hard spend cap | FixRaise the cap, narrow the filter, or accept a provisional verdict when it pauses. |
Symptom or messageQuote credential error, or Compare status no_credential | CauseNo resolvable credential for that provider | FixConnect one under Provider keys; the message names the provider. |
Symptom or messageselect at most 50 traces at a time | CauseSeeding selection over the bound | FixPromote in batches of 50. |
Symptom or messagenone of the N selected traces could be promoted; call PreviewTraceSeeding for the reason on each | CauseEvery selected trace ineligible, commonly content_free_installation or outside_window | FixRun the preview, read each ineligible_reason, and select traces whose bodies were uplinked and are inside retention. |
Symptom or messagean experiment's verdict speaks for its corpus, not for your traffic | CausePromotion attempted from an experiment | FixRun a trial on the scope profile; only a trial verdict can promote. |
Next steps
- Run your first model trial to turn a promising corpus result into evidence that can promote.
- Datasets reference for the full role and reason tables and the trace seeding rules.
- Experiments reference for every quote field and how a run resumes without re-dispatching paid items.
- Compare reference for the streaming event order and why absolute scores are never a delta.
- Judging and verdicts for the thresholds, tie band and judge versioning shared by every eval.