Speculative Decoding
Guess, then verify · draft model and verifier
A speed-up technique where a small, fast 'draft' model guesses several tokens ahead and the large 'target' model checks them in a single forward pass, accepting or rejecting each; done correctly, the output distribution is unchanged.
An LLM generates its answer token by token, reading all of its weights from memory once per new token. On local hardware and at small batch sizes that step is limited by memory bandwidth more than compute: the large model takes roughly as long to process 5 tokens as 1. Speculative decoding exploits that gap.
The loop:
- A draft model (a small model sharing the same tokenizer) quickly guesses the next k tokens.
- The target model (the verifier) scores all k tokens together in one forward pass, computing its own probabilities at each position.
- Draft tokens are checked left to right. A token x the draft proposed with probability q(x), where the target gives p(x), is accepted with probability min(1, p(x)/q(x)). At the first rejection the target samples a replacement from the (normalized) max(0, p − q) distribution and the round ends. If all are accepted, the target adds one more token.
Thanks to this modified rejection sampling, the generated text follows exactly the distribution the target model would produce on its own. Two independent papers showed this: Leviathan, Kalman and Matias (2023), "Fast Inference from Transformers via Speculative Decoding" (ICML 2023; 2–3× faster on T5-XXL with identical outputs), and Chen et al. (2023, DeepMind), "Accelerating Large Language Model Decoding with Speculative Sampling" (2–2.5× on Chinchilla 70B; the distribution is preserved "within hardware numerics"). There is no quality loss; the gain is purely speed and depends on how often the draft is right (the acceptance rate, α). Per Leviathan et al., the expected number of tokens per target pass is (1 − α^(k+1)) / (1 − α).
Variants without a separate draft model:
- Medusa (Cai et al., 2024, "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads"): extra decoding heads are trained on top of the target model and candidates are verified as a tree. The paper reports over 2.2× for Medusa-1 and 2.3–3.6× for Medusa-2. Its "typical acceptance" option aims to keep quality rather than exact distributional equality.
- EAGLE (Li et al., 2024, "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"): a lightweight draft layer predicts the target's second-to-top-layer features instead of tokens; 2.7–3.5× latency speedup on LLaMA2-Chat 70B while preserving the distribution. EAGLE-3 followed in 2025.
- N-gram / prompt lookup: the text itself is the draft; if the last few tokens already appeared in the prompt or history, the tokens that followed them become candidates. Apoorv Saxena's 2023 "Prompt Lookup Decoding" repository popularized the idea. Effective for code edits, summaries and RAG, where output copies a lot from input, with no extra model.
- MTP (multi-token prediction): some models are trained with built-in heads that predict several tokens; vLLM and llama.cpp can use them as the draft.
In tools (per their docs, November 2026):
- llama.cpp (
llama-server):-md/--spec-draft-modelfor the draft model,--spec-typefor the method (draft-simple,draft-eagle3,draft-mtp,ngram-simple,ngram-mod…; defaultnone),--spec-draft-n-maxfor draft tokens per round (default 3). The old--draft-max/--draft-minflags have been removed. - vLLM: the
--speculative-configJSON;methodselectsdraft_model,ngram,mtp,suffix, EAGLE and others, andnum_speculative_tokenssets drafts per round. - MLX-LM (Apple Silicon):
--draft-modeland--num-draft-tokens(default 3) formlx_lm.generate.
For a local setup see the run LLMs locally guide; for the basics see inference and KV cache.
Picture a senior editor and a fast-typing intern. The intern guesses the next five words of the sentence and types them; the editor checks all five at a glance. If the first three are what the editor would have written, they approve them; at the fourth they say "no, I'd say this", write their own word, and the intern carries on from there.
The final text is the same as if the editor had written it alone, but instead of thinking about every word one at a time, the editor usually approves several at once. The better the intern guesses, the faster it goes; if the intern is always wrong, the editor's reading is wasted.
With an acceptance rate α = 0.8 and k = 4 draft tokens per round, Leviathan et al.'s formula gives (1 − 0.8⁵) / (1 − 0.8) ≈ 3.4 tokens per target pass on average, versus 1 in normal decoding. The real speed-up is lower, because the draft model also takes time to run and rejected tokens are wasted.
In a local coding assistant you pair a large model with a much smaller model from the same family. In code completion the next tokens are often predictable (brackets, variable names, repeated patterns), so the acceptance rate is high. With the same setup, a creative, high-temperature story request may gain noticeably less.
For "rename this function"-style requests that rewrite a file, you may not need an extra model at all: n-gram/prompt lookup proposes most of the answer by copying from the input.
Find out how much you gain by measuring, not guessing: in
llama-server's /metrics output, the
llamacpp:spec_decode_num_accepted_tokens_total and
llamacpp:spec_decode_num_draft_tokens_total counters give you the
acceptance rate.
# A separate draft model (a small model sharing the tokenizer)
llama-server -m target-model.gguf \
-md draft-model.gguf --spec-type draft-simple \
--spec-draft-n-max 8
# No draft model: predict from n-grams in the history
llama-server -m target-model.gguf --spec-type ngram-simple --spec-draft-n-max 64# vLLM: speculative decoding from n-gram matches in the prompt
vllm serve <target-model> --speculative-config '{
"method": "ngram",
"num_speculative_tokens": 4,
"prompt_lookup_min": 2,
"prompt_lookup_max": 5
}'
# MLX-LM (Apple Silicon): generate with a draft model
mlx_lm.generate --model <target-model> --draft-model <draft-model> \
--num-draft-tokens 3 --prompt "Hello"- Single-user local inference where generation speed (tok/s) is the bottleneck
- You have a small sibling model with the same tokenizer, or a ready-made EAGLE/MTP draft
- Predictable or input-copying output: code completion, edits, summaries, RAG
- You want lower latency without giving up output quality
- VRAM is already tight; the draft model needs memory for its weights and its own KV cache
- High-concurrency servers; vLLM's docs show model-based methods gain less at high QPS
- The draft and target tokenizers don't match and your tool has no dedicated path for that
Assuming every method is lossless
Standard rejection sampling preserves the distribution. vLLM's docs state plainly that relaxed verification such as FLy is not distribution preserving, and Medusa's typical acceptance doesn't guarantee exact equality either.
Picking too large a draft model
The closer the draft is to the target, the higher the acceptance rate, but the slower the draft itself. Try a few sizes from the same family and find the balance by measuring.
Not measuring the acceptance rate
At a low acceptance rate speculative decoding can be slower than normal decoding. Measure with llama.cpp's metric counters or vLLM's per-request acceptance metrics.
Mistaking floating-point noise for a bug
In theory the distribution is identical, but hardware numerics mean token logprobs may not match exactly. vLLM states it does not guarantee stable logprobs.