Speeding Up Local LLMs for Free: Speculative Decoding Made Simple

Speculative decoding speeds up local LLM generation for free by having a small fast draft model propose several tokens at once while the large target model verifies the entire batch in a single forward pass and accepts only the correct prefix before continuing normally, all without any loss in output quality.

In short: The technique works because a small draft model generates guesses rapidly and the target model checks them efficiently in one go, keeping every accepted token fully consistent with its own decisions.

Why is making tokens one at a time slow?#

Large language models create text autoregressively, predicting exactly one new token during each forward pass through the entire network. Every prediction depends on the full sequence of tokens already chosen, so the model cannot generate future tokens in parallel. On local hardware this creates a severe bottleneck. Each forward pass must load billions of model parameters from VRAM or system memory and perform trillions of arithmetic operations. Even highly optimized engines on consumer GPUs typically achieve only a few tokens per second once models reach 7B parameters or larger. The sequential nature means the expensive weight-loading and matrix-multiplication work repeats for every single token rather than being amortized across many tokens at once. Memory bandwidth limitations often dominate over raw compute, turning what should feel like fluid conversation into noticeable pauses during longer responses or interactive use.

How does speculative decoding speed up for free?#

The key idea is to let a much smaller, faster draft model handle the repetitive work of guessing what comes next, while the big target model only performs verification on groups of those guesses.

A helpful beginner analogy is a busy senior editor paired with a quick intern. The intern rapidly drafts the next several paragraphs based on the current text. The editor then reviews the whole block in one efficient pass, keeps every sentence that matches her standards, and begins rewriting herself only from the first inaccurate sentence onward. The final article remains exactly what the editor would have written alone, yet she finishes much sooner because she avoided rewriting the parts the intern already got right.

In speculative decoding the draft model acts as the intern by quickly producing four to eight or more candidate tokens in a row. These candidates are appended to the current context and fed to the target model for a single forward pass that computes output distributions for every position in the drafted block simultaneously. Verification then proceeds left to right: each drafted token is accepted if it is consistent with the target model's own distribution. Modern implementations use speculative sampling with rejection steps so that the probability of any accepted sequence exactly matches what the target model would have produced on its own. The instant a drafted token fails this check, the remaining guesses are discarded and the target model generates the next token normally from that point. Because accepted tokens were explicitly verified against the target and every correction point triggers genuine target-model generation, the complete final output follows precisely the same distribution as standard decoding from the target model alone. The verify step therefore guarantees zero quality loss; it merely skips redundant full-model computation on the tokens the draft model guessed correctly.

The table also covers two other approaches that reduce redundant work. Standard decoding is the reliable baseline supported everywhere; it simply generates one token after another with no shortcuts. Prompt lookup decoding instead scans the input and recent context for repeated n-gram patterns or boilerplate phrases and reuses matching spans directly when they align with likely continuations. It delivers large gains on highly repetitive material such as code, configuration files, or templated documents, yet still preserves exact output quality because reused spans are validated against the model's predictions or the system falls back to normal generation. All three methods therefore trade extra bookkeeping for fewer full forward passes on the expensive target model while keeping the mathematical behavior of the original model unchanged.

When should you use it locally?#

Speculative decoding delivers its best results when certain practical conditions are met on personal hardware.

1. Confirm you have enough spare VRAM to load both the full target model and the smaller draft model at the same time, because the drafting and verification steps temporarily increase peak memory usage.
2. Pair the target with a small model from the exact same family as the draft, for example a 1B or 3B parameter model alongside a 7B or 13B target, since shared architecture and training data produce higher acceptance rates and therefore larger real-world speedups.
3. Apply the technique to predictable or structured tasks such as writing code, emitting JSON or YAML, filling templates, or generating repetitive technical content where the draft model is likely to guess many tokens correctly before any correction is needed.
4. Favor medium-length or longer generations rather than one- or two-sentence replies, because the time saved by accepting multiple verified tokens per pass compounds across the output and easily outweighs the modest cost of running the draft model.
5. Always benchmark in your own environment using frameworks such as llama.cpp or vLLM, since actual gains vary with GPU or CPU characteristics, the specific model pair, prompt style, and achieved acceptance rate.

Following these guidelines lets you enjoy noticeably more responsive local inference while the final answers remain statistically identical to those produced by the large model running alone.

Note: Reference date 2026-07-12 KST. This explanation reflects current understanding of speculative decoding techniques in popular local inference frameworks. Real speedups depend heavily on your specific hardware configuration, the chosen draft and target model pair, and the acceptance rate achieved on your prompts, so the best approach is always to measure performance directly in your own environment before relying on estimated gains.

Reference links

• Speculative decoding paper (Leviathan et al.)
• Speculative sampling (DeepMind)
• llama.cpp
• vLLM