How Mixture of Experts lowers the compute costs of LLM production

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

In the canonical overview of model architectures we saw in the rise of small language models how more compact parameter sets push down operational costs through targeted task specialization. Where compact dense models simply contain fewer parameters across the board, the Mixture of Experts architecture (MoE) attacks the fundamental scaling problem along a different axis: conditional activation. Instead of consulting every weight in the network for each token generated, a sparse MoE layer activates only a small fraction of the total parameters per compute step.

In production systems, the shift toward MoE has become the dominant strategy for keeping large-scale reasoning capacity affordable. The promise sounds attractive: the performance and knowledge storage of a network with hundreds of billions of parameters, combined with the compute latency and FLOPS requirements of a far smaller model. Even so, this conditional compute method brings a specific technical trade-off with it. The savings on Floating Point Operations per Second (FLOPS) shift the bottlenecks straight to memory bandwidth, VRAM capacity, and network communication between chips.

The structural difference between dense models and sparse architectures

Classic transformer models are dense. When a dense transformer processes a token, that token passes through every parameter in the self-attention layers and in the feed-forward networks (FFN). If a model has 70 billion parameters, all 70 billion weights are loaded and multiplied for every single generated token. This linear coupling between total model size and compute per token forms an economic barrier at large-scale rollout.

A sparse Mixture of Experts architecture replaces the classic FFN layers with a set of several independent experts (usually 8, 16, or more parallel FFN blocks), accompanied by a lightweight routing layer, also called the gating network. The self-attention layers usually stay shared across the whole network in order to capture contextual relations between tokens, but the actual transformation and knowledge application in the FFN layer is assigned dynamically. By engaging only the top-1, top-2, or top-4 experts per token, the model decouples total parameter size (model capacity) from active parameter size (compute load).

The distinction between total and active parameters is essential when budgeting inference. An MoE model with 8x7B parameters in total (roughly 47 billion parameters overall thanks to shared layers) can, for instance, be configured with top-2 routing. While generating a token the system computes with only about 13 billion active parameters. In the learning track on what a token costs and calculating with context length the reader can study how this parameter fraction feeds directly into the theoretical cost price per thousand tokens.

The mathematics of the gating network and routing mechanisms

The heart of a working MoE architecture is the router, or gate. This component decides for each token which experts the representation vectors are forwarded to. Mathematically, the gating network computes a normalized probability distribution over all available experts by applying a softmax function to a linear projection:

# Conceptuele Top-k Routering in PyTorch-stijl
import torch
import torch.nn as nn
import torch.nn.functional as F

class SparseMoERouter(nn.Module):
  def __init__(self, d_model: int, num_experts: int, top_k: int = 2):
    super().__init__()
    self.gate = nn.Linear(d_model, num_experts, bias=False)
    self.top_k = top_k

  def forward(self, x: torch.Tensor):
    # x shape: [batch_size * seq_len, d_model]
    logits = self.gate(x)
    weights, indices = torch.topk(logits, self.top_k, dim=-1)
    routing_weights = F.softmax(weights, dim=-1)
    return routing_weights, indices

Although the mathematics looks simple, the complexity lies in training this router. Without corrections, expert collapse sets in quickly: the network learns to select a few favorite experts for virtually every token, leaving the remaining experts untrained and effectively degrading the system into a smaller dense model. To prevent this, developers introduce auxiliary loss functions during training that enforce an even spread of tokens across all experts.

Modern architectures refine this principle through fine-grained routing. Instead of 8 large experts one might choose 64 or 128 micro-experts, where alongside dynamically selected experts there are also fixed, shared experts that always fire. These shared experts absorb domain-independent grammar and general language patterns, so that the specialized micro-experts can focus purely on specific knowledge domains or syntactic structures.

Why FLOPs drop but VRAM requirements stay the same

The computational efficiency of MoE regularly causes confusion among engineers setting up servers. The reduction in compute (FLOPS) does not automatically translate into lower hardware requirements per server node. To run a model at all, every parameter must be loaded into fast GPU memory (VRAM), including both the active and the inactive experts.

The table shows the clear dividing line: at runtime an 8x7B MoE model needs as much memory as a dense variant of 47 billion parameters, yet per token it requires only the compute of a 13B model. The gain lies in throughput and energy efficiency per compute cluster. The background article on the hardware race around AI chips and compute analyzes how memory bandwidth (HBM) has as a result become a more important limiting factor than raw tensor cores.

The impact on inference infrastructure and batch processing

Inference at production scale revolves around two key statistics: Time To First Token (TTFT) and inter-token latency (throughput in tokens per second). At small batch sizes, MoE models deliver a dramatic speed-up compared with dense models of comparable knowledge quality, because the processor has to perform far fewer matrix multiplications per token.

As soon as batch size grows at busy API endpoints, however, a specific scaling problem appears: token fragmentation. When a server handles 64 requests simultaneously, the tokens within that batch take different routes. Where at a batch size of 1 only two experts are active, at a batch of 64 virtually every expert in memory is addressed. The compute saving per token remains, but the benefits of memory caching and contiguous data transfer diminish.

For this reason MoE inference requires specialized serving engines such as vLLM, TensorRT-LLM, or SGLang with custom CUDA kernels for sparse matrix operations (such as segmented GEMM). These software layers group incoming tokens dynamically per expert before the computation starts. Without these optimizations, the overhead of scattered memory access causes the theoretical compute gain on paper to evaporate in the practice of the server architecture.

Memory bandwidth versus compute: the memory-bound regime

The generation phase of an LLM (emitting token after token) is fundamentally memory-bandwidth bound. For every individual token the model weights have to be copied from DRAM/HBM memory to the compute cores. With a dense 70B model at batch size 1, 140 GB of data must travel across the memory bus every time to produce one single token.

With an MoE model the GPU processes only the weights of the activated experts and the fixed layers. For an 8x7B model with top-2 routing the memory controller has to transfer only about 26 GB of weights per token instead of 95 GB. This explains directly why MoE models reach such high generation speeds on individual streams: the wait on memory transport drops by a factor of three to four.

In production scenarios where compute capacity has to be optimized further, MoE architectures are often combined with application layers for storage and reuse. The document on applying LLM caching effectively explains how semantic and prompt caching bypass the routing overhead entirely for repeated requests, lowering the eventual infrastructure costs even further.

Distributing across multiple GPUs: Expert Parallelism

When an MoE model is too large for a single graphics card, a specific parallelization technique is introduced alongside Tensor Parallelism (TP) and Pipeline Parallelism (PP): Expert Parallelism (EP). Here different experts are distributed across different physical GPUs within or between servers.

[Token Invoer] ---> [Gedeelde Self-Attention op alle GPU's]
                           |
                           v
              [Routeringslaag (Top-2 Selectie)]
              /                               \
    (Token A naar Expert 1)         (Token B naar Expert 7)
            |                               |
            v                               v
    [GPU 0: Expert 1..2]            [GPU 3: Expert 7..8]
            \                               /
             ---> [All-to-All Communicatie] <---
                           |
                           v
            [Gedeelde Output Lagen / Volgende Token]

Expert Parallelism introduceert echter een zware afhankelijkheid van netwerkbandbreedte. Tussen de attention-laag en de experts moeten tokens via zogeheten All-to-All collectieve operaties worden uitgewisseld tussen de verschillende GPU's. Als de interconnects (zoals NVLink binnen een node, of InfiniBand/RoCE tussen nodes) niet snel genoeg zijn, blokkeren de GPU-rekenkernen in afwachting van netwerkpakketten. Waar een dense model vooral rekent en af en toe tensor-synchronisaties uitvoert, vereist een gedistribueerd MoE-systeem een datacenternetwerk met extreem lage latentie en hoge doorvoer.

De trade-offs: complexiteit, load balancing en fijnmazig afstemmen

Ondanks de grote efficiëntievoordelen kent de inzet van Mixture of Experts in een productieomgeving significante nadelen die zorgvuldig moeten worden afgewogen:

• Instabiliteit tijdens fine-tuning: Het trainen of finetunen van MoE-modellen met LoRA of full-parameter fine-tuning is aanmerkelijk complexer dan bij dichte netwerken. De routeringsbeslissingen kunnen verschuiven door taakspecifieke data, waardoor bepaalde experts overbelast raken en het model zijn generalisatievermogen verliest.
• Onvoorspelbaar geheugengebruik bij piekbelasting: Als een plotselinge stroom van gebruikersvragen toevallig tokens genereert die allemaal naar dezelfde specifieke expert worden gerouteerd (load imbalance), ontstaat er een hotspot op de GPU die die specifieke expert host. De verwerkingsrij op die GPU loopt vol, terwijl andere GPU's in het cluster nietsdoend wachten.
• Kwantisatie-uitdagingen: Het kwantiseren van MoE-modellen (bijvoorbeeld naar INT4 of FP4) vereist specialistische kalibratie. Omdat verschillende experts verschillende activatiedynamieken en uitschieters (outliers) vertonen, leidt uniforme kwantisatie vaak tot groter kwaliteitsverlies dan bij homogene dichte netwerken.
• Vaste geheugenvoetafdruk: Organisaties die lokaal draaien hebben nog steeds dure servers nodig met voldoende totale VRAM-capaciteit om het volledige model in te laden, zelfs als de gemiddelde GPU-belasting laag blijft.

De adoptie van deze techniek staat niet op zichzelf; in het bredere ecosysteem zien we dat de ontwikkelingen in open-source LLM-trends ervoor zorgen dat steeds meer open gewichten beschikbaar komen als MoE. Hierdoor wordt hoogwaardige redeneercapaciteit bereikbaar voor organisaties met een beperkt rekenbudget, mits hun hardwarearchitectuur is ingericht op de bijbehorende geheugeneisen.

Kostenanalyse in de praktijk: wanneer kies je voor MoE?

De keuze tussen een dense architectuur en een sparse MoE-model hangt in de praktijk af van de operationele context waarin het model wordt ingezet:

Kies voor een dense model wanneer de toepassing lokaal draait op edge-apparaten of werkstations met strikt beperkt VRAM (zoals laptops of embedded systems), wanneer batchgroottes constant zeer groot zijn en geheugenbandbreedte maximaal wordt benut, of wanneer continue fine-tuning op kleine domeindatasets een vaste vereiste is binnen het ontwikkelteam.

Kies voor een Mixture of Experts-architectuur wanneer het systeem via een API of centrale serveromgeving opereert met variabele werklasten, wanneer lage latentie per gebruiker essentieel is zonder concessies te doen aan brede achtergrondkennis en redeneervaardigheden, en wanneer de infrastructuur beschikt over voldoende geheugenruimte (eventueel ondersteund door snelle interconnects tussen kaarten).

De economische realiteit van AI-infrastructuur dwingt engineeringteams om scherp te sturen op de kosten per gegenereerd antwoord. Mixture of Experts lost de inherente spanning tussen modelcapaciteit en rekenbudget niet op door magie, maar door een doordachte herverdeling van computetaken over tijd en hardware. Daarmee heeft het een vaste plaats veroverd als hoeksteen van moderne, kostenefficiënte LLM-productiesystemen.