Speculative decoding: accelerating LLM inference

In the canonical overview of model architectures, the article on the rise of small language models (SLMs) already showed that smaller systems offer enormous operational advantages in compute and memory usage. Yet the question remains how we can raise the absolute generation speed of very large language models without compromising output quality. Large language models generate text token by token. Every new token requires a full compute pass over hundreds of billions of parameters, which makes the process fundamentally limited by memory bandwidth rather than raw compute. Speculative decoding resolves this bottleneck by deploying an asymmetric tandem of models.

By letting a compact, fast model run ahead speculatively and having the predicted tokens verified by the large target model in a single parallel step, this technique breaks with the traditional sequential paradigm. In this article we analyze the mathematical foundation, the hardware implications, the practical acceptance rate, and the hard preconditions of speculative inference in production environments.

The memory-bandwidth bottleneck in autoregressive generation

Classical inference with autoregressive language models is heavily memory-bandwidth bound. When a model produces a single token, all the weights of the network have to be moved from slower video memory (VRAM) to the fast compute cores of the processor or graphics card. For a neural network of 70 billion parameters at 16-bit precision, that means roughly 140 gigabytes of weights have to be transferred per token, regardless of whether the word to be generated is complex or a predictable punctuation mark.

The actual compute cores (such as Tensor Cores) complete the required matrix multiplications for a single token in a fraction of the time needed to load those weights. Compute power therefore remains underused for most of the generation cycle. This phenomenon explains why batch processing is more efficient: when several prompts are processed at once, the same weights are reused for dozens of tokens at a time, which raises what is known as arithmetic intensity .

In interactive applications — such as chatbots or code assistants — the batch size for an individual user is one, however. The limiting factor here is purely the succession of consecutive memory transfers. Only when we can evaluate multiple tokens in parallel within a single forward pass does the return on available hardware rise substantially. This hardware question ties directly into the broader dynamic worked out in the overview of the race for AI chips and hardware capacity, which centers on the physical limits of memory interfaces.

The basic principle: draft model and target verification

Speculative decoding breaks through the sequential barrier by splitting the generation process into two separate roles: speculation and verification. The system combines a small, fast neural network (the draft model) with the full, heavy network (the target model). Both models must use the same vocabulary and the same tokenizer to avoid representation errors.

In practice the process runs in fixed iterative cycles:

• Phase 1 (drafting): The small draft model autonomously and sequentially generates a sequence of, say, K candidate tokens (often 3 to 6 tokens). Because the draft model is only a fraction of the size of the target model, this step runs extremely fast with minimal memory transfers.
• Phase 2 (verification): The K candidate tokens are fed to the large target model together with the existing context in a single forward pass. The target model computes the probability distributions for all positions in the sequence simultaneously.
• Phase 3 (acceptance & correction): A statistical acceptance algorithm checks the predicted tokens from left to right. Accepted tokens are finalized immediately. As soon as a token is rejected, the target model generates a corrected replacement token from its own distribution right away, and the remaining speculative tail is discarded.

Anyone who wants to work through the exact conceptual steps and data flows interactively can turn to the interactive guide on how speculative decoding increases generation speed for a didactic treatment of the internal matrix steps. The crucial gain is that a single heavy forward pass of the target model now yields several accepted tokens in exactly the same compute time previously needed for a single token.

Mathematical proof of distribution preservation

A persistent misconception about speculative decoding is that it is an approximation technique that degrades final model quality. This is demonstrably untrue. Provided it is implemented correctly through speculative rejection sampling, the resulting probability distribution of the output is mathematically identical to the distribution the target model would produce on its own.

Let $M_t(x)$ be the probability distribution over the vocabulary generated by the target model for a given position, and $M_d(x)$ the probability distribution of the draft model. When the draft model proposes a token $x$, we accept that token with the following transition probability:

P(accepteer x) = min(1, M_t(x) / M_d(x))

When the proposed token $x$ is rejected (with probability $1 - P(\text{accept } x)$), we draw a new token $x'$ from a normalized residual distribution:

P(x') = max(0, M_t(x') - M_d(x')) / som_over_vocab(max(0, M_t(v) - M_d(v)))

This correction step compensates exactly for the deviations of the draft model. If the probability assigned by the target model is greater than or equal to that of the draft model, acceptance is guaranteed. If the probability is lower, the residual draw ensures that tokens undervalued by the draft model are still chosen with the correct marginal probability. In deterministic generation (greedy decoding at temperature zero), this simply reduces to checking whether the predicted token matches the argmax of the target model.

Hardware efficiency and acceptance rate in practice

The effective speedup achieved in production depends directly on two factors: the acceptance rate ($\alpha$) of the proposed tokens and the relative compute time of the draft model versus the target model. The acceptance rate indicates the average percentage of speculative tokens that pass the verification phase successfully.

In the table below we analyze how different tasks and domains perform under speculative inference at a fixed speculation length ($K=5$):

Measurements show that programming languages and formal documents deliver the highest return. This is because programming language constructs, imports, closing characters, and standard syntax have relatively low information entropy. The draft model predicts such patterns almost flawlessly. With creative prose at a high sampling temperature, by contrast, $\alpha$ drops considerably, which makes the compute gain marginal.

Architectural variants: draft models, Medusa, and EAGLE

The original design of speculative decoding requires two separate neural networks to be loaded into VRAM at the same time. That places an extra claim on the memory budget. Advanced variants have been developed to work around this limitation:

1. Standalone draft models

Here a smaller sibling model from the same family acts as the draft model (an 8B-parameter variant as assistant to a 70B-parameter network, for example). The advantage is modular deployment; the drawback is that the draft model has to reserve its own full KV cache and parameter sets in video memory.

2. Multi-head speculation (Medusa)

Instead of a separate network, multiple parallel decoding heads are trained on top of the last hidden state of the target model. Each additional head predicts a token at position $t+1, t+2, \dots, t+k$. This eliminates the need for a second network and saves considerably on memory overhead, although these heads have to be trained specifically for each model.

3. Dynamic contextual speculation (EAGLE)

EAGLE (Extensible Autoregressive Generation with Loosely-coupled Expansion) improves speculative quality by feeding the speculation head not only earlier tokens but also the hidden states (feature vectors) of the previous layer. That raises the acceptance rate considerably, particularly on more complex reasoning tasks, because semantic context is preserved far more richly during the speculative steps.

For those who want to configure and tune such infrastructure in production themselves, the guide on configuring speculative decoding for faster LLM tokens offers practical configuration examples for runtimes such as vLLM, TensorRT-LLM, and llama.cpp.

Integration with mixture of experts and context caching

Speculative decoding does not stand alone; it forms a powerful synergy with other architectural optimizations. An important point of contact lies with sparse models. As described in detail in the article on how mixture of experts lowers compute costs, an MoE architecture activates only a subset of its total parameter volume per token. When a compact MoE model acts as the draft model, speculation latency drops even further, because the draft model's forward pass requires extremely few floating-point operations.

There is also a direct interaction with the KV cache (key-value cache). Speculative verification requires dynamic tree management of the cache (tree attention). Instead of a linear sequence of tokens, the draft model can present several branching hypotheses to the target model at once through a special attention mask. The target model evaluates this tree structure in a single compute step, selects the longest valid path, and prunes the rejected branches from the memory cache immediately.

Practical implementation: a minimalist verification algorithm

To make the interplay between the draft model and the target model clear, the Python example below shows the core mechanism of autoregressive speculation under deterministic (greedy) inference:

def speculative_step(target_model, draft_model, context, gamma=4):
  # 1. Genereer gamma speculatieve tokens met het snelle model
  draft_tokens = []
  curr_context = list(context)
  
  for _ in range(gamma):
    next_token = draft_model.predict_next(curr_context)
    draft_tokens.append(next_token)
    curr_context.append(next_token)
    
  # 2. Evalueer alle kandidaatposities parallel met het target model
  target_logits = target_model.forward_pass(context + draft_tokens)
  
  # 3. Verifieer de voorspelde reeks
  accepted_tokens = []
  for i, token in enumerate(draft_tokens):
    true_token = target_logits[len(context) + i - 1].argmax()
    if token == true_token:
      accepted_tokens.append(token)
    else:
      # Bij een fout nemen we de correctie van het target model
      accepted_tokens.append(true_token)
      return accepted_tokens  # Breek af bij de eerste mismatch
      
  # 4. Als alle tokens correct waren, voeg een extra bonus token toe
  bonus_token = target_logits[len(context) + gamma - 1].argmax()
  accepted_tokens.append(bonus_token)
  return accepted_tokens

This basic pattern shows why a time gain still arises even with a partial mismatch: as soon as two of the four tokens are correct, for example, the cycle still yields three validated tokens (two approved plus the corrected deviation) within the duration of a single heavy verification step.

Explicit drawbacks, trade-offs, and preconditions

Although speculative decoding delivers substantial speed gains, the technique has specific drawbacks and operational limitations that have to be weighed carefully:

• Memory footprint (VRAM): Loading both the target model and the draft model at the same time requires extra video memory. On systems where the main model barely fits in VRAM, adding a draft model can lead to out-of-memory errors or force more aggressive quantization.
• Reduced throughput at high batch sizes: Speculative decoding is designed for low concurrency (batch size 1 to 4). On large-scale production servers already running continuously with saturated GPU compute cores (high batch sizes), the verification step consumes compute that could otherwise have served other users. In such situations, total server throughput per second can actually fall.
• Software complexity: Frameworks have to support advanced tree attention and non-linear KV cache management. Errors in the implementation of rejection sampling can lead to subtle distribution shifts and quality loss.
• Quantization sensitivity: If the draft model and the target model are quantized asymmetrically (an FP8 target with an INT4 draft, for example), divergence between the two probability distributions can increase, causing the acceptance rate ($\alpha$) to drop sharply.

Conclusion and outlook on inference architectures

Speculative decoding marks a fundamental shift in how inference infrastructure is designed. By working around hardware memory constraints through mathematically pure parallel verification, interactive generation speed doubles in many business use cases without any loss of quality.

Further development is moving toward architectures in which speculation heads are integrated directly during pretraining of the main model, making external draft models unnecessary. For engineers and organizations operating local or private AI environments, speculative decoding is an indispensable link in delivering enterprise-quality language models at low latency on existing hardware infrastructure.