Compute can be spent when the answer is needed, not only when the model is built. But sampling more answers only helps if you can tell which one is right — and that distinction decides everything.
All the compute so far has been spent before the user arrives. But you can also spend it afterwards: sample the same question ten times, or a hundred, and pick the best answer. Whether that helps turns entirely on a question people routinely skip — if a correct answer is somewhere in your hundred samples, how exactly do you intend to find it?
Sample \( k \) independent attempts at a problem, each succeeding with probability \( p \). The probability that at least one succeeds is one minus the probability all fail:
\[ \text{coverage}(k) = 1 - (1-p)^{k}. \]This grows quickly — faster than most people expect, which is exactly what makes the next point so easy to miss. A model that solves a problem one time in twenty covers half of such problems within fourteen attempts. But note carefully what has been computed: the probability that a correct answer is somewhere in the set. Whether you can deliver it depends on whether you can identify it.
Coverage is the most over-quoted number in this part of the field. It measures whether a correct answer exists among the samples, not whether the system returns one. You only realize coverage in production if you have a selector — a verifier, a test suite, a reward model — that picks the right sample. Reporting coverage as though it were accuracy overstates a system by exactly the selector's fallibility, which is often large.
With a selector of reliability \( v \) — the probability it identifies a correct answer when one is present — the delivered accuracy is roughly \( v\bigl[1-(1-p)^k\bigr] \). The selector, not the sampler, is usually the binding constraint.
Small model, many tries, against big model, one try
Compare at equal compute. A small model costs 1 unit per attempt and succeeds with \( p = 0.2 \). A large model costs 10 units and succeeds with \( p = 0.6 \). Budget: 10 units.
The small model gets 10 attempts: coverage \( = 1 - 0.8^{10} = 1 - 0.107 = \) 0.893. The large model gets one: \( 0.6 \). With a perfect selector, sampling wins decisively.
Now introduce a realistic selector at \( v = 0.7 \): delivered accuracy becomes \( 0.7 \times 0.893 = \) 0.625 — barely better than the large model's single attempt. The entire advantage lived in the assumption of perfect selection.
This is why domains with automatic verification — code that must pass tests, mathematics with checkable answers, where \( v \approx 1 \) — led the inference-scaling era. Where verification is hard, the strategy degrades sharply.
(a) At \( p = 0.05 \), how many samples give 50% coverage? (b) Budget 32 units: model A costs 1 with \( p = 0.1 \); model B costs 8 with \( p = 0.45 \) (so four attempts). With a perfect selector, which wins?
(a) \( 0.95^k = 0.5 \Rightarrow k = \ln 0.5/\ln 0.95 = 13.5 \), so 14. (b) A: \( 1 - 0.9^{32} = 0.966 \). B: \( 1 - 0.55^4 = 0.908 \). A wins narrowly.
A cheaper form of selection is to let the samples vote: return the most common answer. Chapter 1 computed the case of five attempts at \( p = 0.6 \), giving \( 0.683 \). The governing fact is a threshold.
Voting helps only above one half
For \( 2m+1 \) independent attempts each correct with probability \( p \) (assuming errors do not coordinate on a single wrong answer), the majority is correct with probability \( \sum_{j=m+1}^{2m+1}\binom{2m+1}{j}p^j(1-p)^{2m+1-j} \). By the law of large numbers this tends to 1 when \( p > \tfrac12 \) and to 0 when \( p < \tfrac12 \), as the number of attempts grows.
The failure case is worth computing once. At \( p = 0.4 \) with five attempts, symmetry gives majority accuracy \( 1 - 0.683 = \) 0.317 — worse than a single attempt's \( 0.4 \). Below the threshold, voting reliably amplifies the error rather than correcting it.
The most important distinction in this chapter is not arithmetic but conceptual, and it is the one worth carrying away. Sampling many times and selecting redistributes probability mass the model already had. It surfaces a capability; it does not add one. If the model's per-attempt probability of solving a class of problem is zero, no amount of sampling produces a solution, and no verifier can select one from a set that contains none.
Training a model to reason at length — using verifiable rewards and the group-relative method of Chapter 14 — is different in kind. It changes the weights, and therefore changes \( p \) itself. That mechanism can create a capability. Prompting a model to think step by step, at fixed weights, elicits; training it on verified reasoning traces creates.
Classify each as eliciting or creating: (a) best-of-64 sampling with unit tests; (b) majority voting; (c) reinforcement training on verified mathematics; (d) a longer step-by-step prompt at fixed weights.
(a) elicit — selects from what the model already produces. (b) elicit. (c) create — the weights change, so \( p \) itself changes. (d) elicit — no weight change; it changes which of the model's existing behaviors is expressed.
Coverage grows as \( 1-(1-p)^k \) and is not accuracy; the selector's reliability multiplies it, and in domains without automatic verification that multiplier dominates. Voting is a cheap selector with a hard threshold at one half, below which it actively hurts. And the elicit–create distinction separates the mechanisms that surface existing ability from the one that manufactures it. Chapter 16 leaves language entirely for the era's other generative pillar, where an image is produced by reversing a controlled corruption.
A · Drills
B · Problems
C · Challenge
Reproduce the coverage formula, the matched-compute comparison template, and Derivation 15.1's threshold on blank paper. Attempt B-1 closed book, and defend each classification in Drill 15.2 with one sentence. You pass when you never again quote a coverage figure without naming the selector.