AI Atlas
Advanced · ~4 min read #speculative-decoding #inference #latency

Speculative Decoding

Guess, then verify · draft model and verifier

A speed-up technique where a small, fast 'draft' model guesses several tokens ahead and the large 'target' model checks them in a single forward pass, accepting or rejecting each; done correctly, the output distribution is unchanged.

GUESS, THEN VERIFY IN ONE PASSDraft modelsmall · fastdefadd(a,b):passk = 5target scores all of them in one passTarget modellarge · verifierdef✓add✓(a,✓b):✗pass—Resultdefadd(a,b,at the rejection the targetsamples its own tokenLARGE-MODEL PASSES · 4 TOKENSnormalspeculative1 pass + draftaccept: min(1, p/q)output distribution unchangedLeviathan et al. 2023 · Chen et al. 2023
Definition

An LLM generates its answer token by token, reading all of its weights from memory once per new token. On local hardware and at small batch sizes that step is limited by memory bandwidth more than compute: the large model takes roughly as long to process 5 tokens as 1. Speculative decoding exploits that gap.

The loop:

  1. A draft model (a small model sharing the same tokenizer) quickly guesses the next k tokens.
  2. The target model (the verifier) scores all k tokens together in one forward pass, computing its own probabilities at each position.
  3. Draft tokens are checked left to right. A token x the draft proposed with probability q(x), where the target gives p(x), is accepted with probability min(1, p(x)/q(x)). At the first rejection the target samples a replacement from the (normalized) max(0, p − q) distribution and the round ends. If all are accepted, the target adds one more token.

Thanks to this modified rejection sampling, the generated text follows exactly the distribution the target model would produce on its own. Two independent papers showed this: Leviathan, Kalman and Matias (2023), "Fast Inference from Transformers via Speculative Decoding" (ICML 2023; 2–3× faster on T5-XXL with identical outputs), and Chen et al. (2023, DeepMind), "Accelerating Large Language Model Decoding with Speculative Sampling" (2–2.5× on Chinchilla 70B; the distribution is preserved "within hardware numerics"). There is no quality loss; the gain is purely speed and depends on how often the draft is right (the acceptance rate, α). Per Leviathan et al., the expected number of tokens per target pass is (1 − α^(k+1)) / (1 − α).

Variants without a separate draft model:

  • Medusa (Cai et al., 2024, "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads"): extra decoding heads are trained on top of the target model and candidates are verified as a tree. The paper reports over 2.2× for Medusa-1 and 2.3–3.6× for Medusa-2. Its "typical acceptance" option aims to keep quality rather than exact distributional equality.
  • EAGLE (Li et al., 2024, "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"): a lightweight draft layer predicts the target's second-to-top-layer features instead of tokens; 2.7–3.5× latency speedup on LLaMA2-Chat 70B while preserving the distribution. EAGLE-3 followed in 2025.
  • N-gram / prompt lookup: the text itself is the draft; if the last few tokens already appeared in the prompt or history, the tokens that followed them become candidates. Apoorv Saxena's 2023 "Prompt Lookup Decoding" repository popularized the idea. Effective for code edits, summaries and RAG, where output copies a lot from input, with no extra model.
  • MTP (multi-token prediction): some models are trained with built-in heads that predict several tokens; vLLM and llama.cpp can use them as the draft.

In tools (per their docs, November 2026):

  • llama.cpp (llama-server): -md / --spec-draft-model for the draft model, --spec-type for the method (draft-simple, draft-eagle3, draft-mtp, ngram-simple, ngram-mod …; default none), --spec-draft-n-max for draft tokens per round (default 3). The old --draft-max / --draft-min flags have been removed.
  • vLLM: the --speculative-config JSON; method selects draft_model, ngram, mtp, suffix, EAGLE and others, and num_speculative_tokens sets drafts per round.
  • MLX-LM (Apple Silicon): --draft-model and --num-draft-tokens (default 3) for mlx_lm.generate.

For a local setup see the run LLMs locally guide; for the basics see inference and KV cache.

Analogy

Picture a senior editor and a fast-typing intern. The intern guesses the next five words of the sentence and types them; the editor checks all five at a glance. If the first three are what the editor would have written, they approve them; at the fourth they say "no, I'd say this", write their own word, and the intern carries on from there.

The final text is the same as if the editor had written it alone, but instead of thinking about every word one at a time, the editor usually approves several at once. The better the intern guesses, the faster it goes; if the intern is always wrong, the editor's reading is wasted.

Real-world example

With an acceptance rate α = 0.8 and k = 4 draft tokens per round, Leviathan et al.'s formula gives (1 − 0.8⁵) / (1 − 0.8) ≈ 3.4 tokens per target pass on average, versus 1 in normal decoding. The real speed-up is lower, because the draft model also takes time to run and rejected tokens are wasted.

In a local coding assistant you pair a large model with a much smaller model from the same family. In code completion the next tokens are often predictable (brackets, variable names, repeated patterns), so the acceptance rate is high. With the same setup, a creative, high-temperature story request may gain noticeably less.

For "rename this function"-style requests that rewrite a file, you may not need an extra model at all: n-gram/prompt lookup proposes most of the answer by copying from the input.

Find out how much you gain by measuring, not guessing: in llama-server's /metrics output, the llamacpp:spec_decode_num_accepted_tokens_total and llamacpp:spec_decode_num_draft_tokens_total counters give you the acceptance rate.

Code examples
llama.cpp: with a draft model and with n-grams bash
# A separate draft model (a small model sharing the tokenizer)
llama-server -m target-model.gguf \
  -md draft-model.gguf --spec-type draft-simple \
  --spec-draft-n-max 8

# No draft model: predict from n-grams in the history
llama-server -m target-model.gguf --spec-type ngram-simple --spec-draft-n-max 64
vLLM and MLX-LM bash
# vLLM: speculative decoding from n-gram matches in the prompt
vllm serve <target-model> --speculative-config '{
  "method": "ngram",
  "num_speculative_tokens": 4,
  "prompt_lookup_min": 2,
  "prompt_lookup_max": 5
}'

# MLX-LM (Apple Silicon): generate with a draft model
mlx_lm.generate --model <target-model> --draft-model <draft-model> \
  --num-draft-tokens 3 --prompt "Hello"
When to use
  • Single-user local inference where generation speed (tok/s) is the bottleneck
  • You have a small sibling model with the same tokenizer, or a ready-made EAGLE/MTP draft
  • Predictable or input-copying output: code completion, edits, summaries, RAG
  • You want lower latency without giving up output quality
When not to use
  • VRAM is already tight; the draft model needs memory for its weights and its own KV cache
  • High-concurrency servers; vLLM's docs show model-based methods gain less at high QPS
  • The draft and target tokenizers don't match and your tool has no dedicated path for that
Common pitfalls

Assuming every method is lossless

Standard rejection sampling preserves the distribution. vLLM's docs state plainly that relaxed verification such as FLy is not distribution preserving, and Medusa's typical acceptance doesn't guarantee exact equality either.

Picking too large a draft model

The closer the draft is to the target, the higher the acceptance rate, but the slower the draft itself. Try a few sizes from the same family and find the balance by measuring.

Not measuring the acceptance rate

At a low acceptance rate speculative decoding can be slower than normal decoding. Measure with llama.cpp's metric counters or vLLM's per-request acceptance metrics.

Mistaking floating-point noise for a bug

In theory the distribution is identical, but hardware numerics mean token logprobs may not match exactly. vLLM states it does not guarantee stable logprobs.