A golden dataset is a small, versioned, hand-curated set of input-output pairs that defines what "working" means for your AI feature. Paired with regression suites in CI, it converts prompt changes from vibes into engineering — the same conversion unit tests gave software.
The need arrives with the first incident: a "small prompt tweak" that improved one example and silently broke a dozen others nobody checked. LLM outputs are non-deterministic and multi-dimensional, so ad-hoc eyeballing scales worse than it does for traditional code — yet the change cadence is faster. The only stable ground is a fixed evaluation set: real production inputs (anonymized), expected outputs or rubric criteria, difficulty strata, and the known failure modes captured as permanent fixtures — every production incident becomes a test case, the same discipline bug reports gave software.
The harness matters as much as the data. Assertions split into two families: deterministic checks (JSON schema validity, required fields, no PII, citations resolve to real sources) that belong in CI's fast lane, and model-gradedrubrics (helpfulness, tone, correctness against a reference) that need their own calibration — fixed judge prompts, spot-checked against human labels, and never the same model as the one under test. Evaluation runs on every prompt or model change, with results diffed against the previous release and gated like any other build.
Anti-Patterns to Refuse
- Teaching to the test: if prompts get patched until the golden set passes, the set is spent — hold out a rotating slice the team never sees.
- Frozen forever: golden sets drift stale as the product evolves; schedule quarterly curation the way you schedule dependency upgrades.
- Average-only reporting: a 2% aggregate regression can hide a 40% regression on the hardest slice — report per-stratum, always.
- Judge drift: if the grader prompt changed, historical scores are not comparable — version graders like code.
Teams sometimes resist the ritual as overhead — until the first model deprecation lands. The vendor will retire your model with three months' notice; the golden suite is how you find out what breaks before your users do.