Blog

Speculative decoding

Deval, with AI-assisted editing

The small AI is allowed to be wrong. Its guesses never become final text until the large AI checks them.

When the guesses are good, one expensive check can return several pieces of text. The large AI still decides what is allowed.

That is speculative decoding.

Why text generation is slow

A language modelA system that predicts the next piece of text from the text it has already seen. writes one tokenA small piece of text. It can be a word, part of a word, or punctuation. at a time.

It cannot choose token 2 until token 1 exists. It cannot choose token 3 until token 2 exists. The work forms a line:

large model -> "The"
large model -> "cat"
large model -> "sat"
large model -> "nearby"

Servers remember earlier calculations, so the model does not start over for every token. But the waiting remains. Each new token depends on the one before it.

Guess first, check together

Speculative decoding adds a draft modelThe small, fast model that guesses a few tokens ahead.. This small model guesses several tokens first.

The target modelThe large model whose output probabilities must be preserved. then checks those positions in one pass. Modern AI hardware is good at doing similar calculations together. This is called parallel workDoing several calculations at the same time instead of waiting for each one to finish..

Four waits become one check

The small model does extra cheap work so the large model can check several positions together.

Normal decoding compared with speculative decodingNormal decoding uses four serial large-model passes to produce four tokens. Speculative decoding uses a small model to guess four tokens, then checks those positions together in one large-model pass.Normal decoding4 large-model passeslarge modelwrites "The"large modelwrites "cat"large modelwrites "sat"large modelwrites "nearby"Speculative decoding1 large-model passsmall modelguesses aheadThecatsatdownlarge model checks all four positions together
The small model still guesses one token at a time. The saving comes from using fewer serial passes through the large model.

The draft adds work. That sounds backward, but the added work is cheap. It can save time by reducing how often the large model must run in sequence.

What happens to a bad guess

The small model guesses:

The cat sat down

The large model prefers:

The cat sat nearby

Tap the verbs to see the sentence change.

The first bad guess gets replaced

Tap each verb. The sentence changes as the round moves.

One speculative decoding roundThe small model guesses four tokens: The cat sat down.small modelguessesThecatsatdownlarge modelkeeps the accepted prefixRound outputThe cat sat nearby4 tokens from 1 large-model check

The small model guesses four tokens: The cat sat down.

Checking stops at the first failed guess. Any later guesses would depend on text the large model did not accept, so the system throws them away.

The system reads the guesses from left to right. It keeps the good prefixThe accepted tokens from the start of a proposed sequence., which means the accepted tokens at the start.

It stops at the first bad guess. That position now needs a new token. The repair stepThe step that replaces a rejected guess using chances the small model missed. picks it. Any guess after it is thrown away because those later guesses were based on text the large model rejected.

If every guess passes, the large model can add one more token. It already calculated the next position while checking the draft.

Why the repair is needed

In the previous section, the system rejected down. That leaves an empty slot after The cat sat.

The system cannot simply ask the small model again. The small model guesses some words too often and others too rarely. It also cannot make a fresh normal pick with the large model. The accepted guesses have already used some of the large model's chances.

That is the repair's job. It fills the empty slot using only the chances the small model missed. This lets the shortcut save work without changing the large model's original chances.

The large model does not have one fixed next word. For one position, picture its chances as 100 points:

  • 50 points for cat
  • 30 points for dog
  • 20 points for owl

The small model uses the same 100 points, but gets the split wrong. It gives 60 to cat, 10 to dog, and 30 to owl.

The check keeps only the points that fit inside the large model's split:

  • It keeps 50 cat points. The extra 10 do not fit.
  • It keeps 10 dog points. That is all the small model supplied.
  • It keeps 20 owl points. The extra 10 do not fit.

Only 80 points remain. If the system stopped here, dog would be too rare. The small model would have changed the answer.

The system cannot make a fresh 50, 30, 20 pick for the empty space. cat and owl already have all their points. Only dog is short. The repair adds the missing 20 dog points.

The check leaves a hole. Repair fills it.

The small model offered too few dog points. The repair supplies the missing 20.

The check leaves a gap and the repair fills itThe large model gives cat 50 percent, dog 30 percent, and owl 20 percent. The check keeps 50 percent cat, 10 percent dog, and 20 percent owl from the small model. That leaves a 20 percent gap. The repair fills it with dog, which restores the large model's original chances.Targetcat 50%dog 30%owl 20%Keptcat 50%10%owl 20%Repairdog 20%Finalcat 50%dog 30%owl 20%
The real system uses decimal probabilities. The 100 points make the same math easier to see.

The final split is back to 50 cat, 30 dog, and 20 owl. The technical name for this list of chances is a probability distributionA list of possible outcomes and the chance assigned to each one..

Without the repair, the small model would change which words appear. With the repair, the shortcut changes the work, not the large model's chances. That is what "exact" means here.

The system then makes one random pick from those chances. This is called samplingRandomly choosing a token using the chances assigned by the model.. Two runs can pick different words. Across many runs, each word appears at the large model's original rate.

Show the short proof

Call the large model's probability p and the small model's probability q. A guessed token is kept with chance min(1, p/q).

kept probability = q * min(1, p/q)
                 = min(q, p)

The repair samples only from the part the small model missed:

missing probability = max(0, p - q)

Put the kept part and the missing part together, and the result is p.

When it is faster

Speculative decoding usually helps when:

  • the draft model is much cheaper than the target model;
  • the draft and target agree often;
  • the hardware can check a short group of positions efficiently.

It can be slower when the first guess often fails, the draft model is too large, or moving data between models costs more than the saved target passes.

Serving many users adds another tradeoff. A setup can improve latencyHow long one request waits for output. for one request while hurting throughputHow much total output the system produces across all requests. for the whole server. Both need to be measured.

Measured speedups

Leviathan, Kalman, and Matias measured about 2x to 3x faster decoding on T5-XXL in their test setups.

Chen and coauthors measured about 2x to 2.5x faster decoding in their Chinchilla experiments.

Those results show that the method can work. They are not a promise for every model or server. The result depends on the two models, the hardware, how many requests run together, the input text, and how the model picks each token.

The full loop

  • A small model guesses ahead.
  • The large model checks the guesses together.
  • The first bad guess ends the accepted prefix.
  • A correction keeps the large model's original probability distribution.
  • The method is useful only when the cheap guesses save more time than they cost.

Glossary

Token
A small piece of text. It can be a whole word, part of a word, or punctuation.
Draft model
The small, fast model that guesses a few tokens ahead.
Target model
The large model whose output probabilities must be preserved.
Prefix
The accepted text from the start of the proposed sequence.
Latency
How long you wait for the model to produce output.
Throughput
How much total output a system produces across all requests.

Sources