Chain-of-thought distillation trains a small, cheap model on the reasoning traces of a large, slow one — capturing much of the teacher's capability at a fraction of the inference cost. It is how frontier reasoning gets compressed into products that can afford to ship.
The recipe has three steps. Generate: prompt a frontier model to solve tens or hundreds of thousands of problems, keeping only traces that verify (correct final answers, tests passing). Filter: quality over quantity — traces are deduplicated, difficulty- stratified, and stripped of lucky guesses. Fine-tune: train the student model to produce those traces given only the questions. DeepSeek-R1's public release of 800k curated reasoning traces, and the wave of sub-10B models trained on them that suddenly solved competition math, made the pattern impossible to ignore: the traces, not the architecture, carried much of the capability.
The honest caveats cluster around what distillation cannot transfer. Students learn the teacher's outputs, not its self-correction — traces are post-hoc rationalized and omit the hesitation, backtracking, and dead ends where real reasoning lives. Students also inherit teacher blind spots amplified by filter selection, and generalization beyond the distribution of traced problems remains sharply limited: a student distilled on math traces will reason about math, and regress elsewhere.
Making Distillation Pay
- Verify at generation time: only traces whose final answers check out enter the corpus — garbage traces teach confident garbage.
- Stratify deliberately: oversample the failure modes your production traffic actually exhibits, not the benchmark's favorites.
- Route, don't replace: deploy the student as the default tier and escalate to the teacher on low confidence — the combo beats either alone on cost-adjusted accuracy.
- Re-distill on a cadence: the teacher improves; a student frozen last quarter is a depreciating asset.
Strategically, distillation is how the frontier propagates: each capability demonstrated at the top becomes affordable at the bottom within quarters. The teams that win with it treat trace curation — not model training — as the core competency.