Self-consistency is embarrassingly simple and surprisingly hard to beat: sample the same reasoning problem k times at temperature, then pick the answer that appears most often. The majority vote over reasoning paths filters out the errors that any single path accumulates.
The insight (Wang et al., 2022) is that chain-of-thought decoding is greedy — it commits to one path through a reasoning space where many valid paths exist. By sampling k diverse paths and marginalizing over them, the correct answer tends to be reached by more distinct routes than any specific wrong answer, because errors are largely path-specific while truths are path-independent. On GSM8K math, the technique added roughly 10–20 absolute points over single-path CoT with the same base model — no fine-tuning, no tooling, just arithmetic over samples.
The cost model is straightforward — roughly k× tokens — which makes budget allocation the entire engineering question. Accuracy gains are concave: the jump from 1 to 5 samples dwarfs the jump from 20 to 40. Sampling parameters matter too: enough temperature for genuine diversity, but not so much that paths degenerate. And the vote needs care — answers must be normalized into equivalence classes (0.5, 1/2, and fifty percent must count as the same vote) before majority is meaningful.
When to Reach for It
- Verifiable end-states: math, extraction, classification — tasks whose answers can be compared across paths. Open-ended prose has no majority to vote on.
- High-stakes, latency-tolerant calls: offline grading, report generation, nightly pipelines — anywhere accuracy dominates response time.
- Confidence estimation for free: the vote share is a usable uncertainty signal — route low-agreement cases to a stronger model or a human.
Self-consistency also deserves credit as the intellectual ancestor of an entire research direction: verify, sample, and aggregate at inference time. It is the cheapest member of the test-time-compute family — the first thing to try before reaching for trees, verifiers, or longer thinking budgets.