|
Stop compromising on AppSec (Sponsor)
 |
Checkmarx Fusion combines hybrid rules and AI reasoning to catch complex zero day bugs in AI generated code without the noise.
For an industry spending billions of dollars on increasingly powerful GPUs, one of the strangest things about modern AI inference is how often those GPUs are not limited by their ability to calculate.
They are limited by their ability to move data.
When a large language model generates a response, the apparent simplicity of text streaming hides an expensive process underneath. The model generates one token, updates its context, generates another token, updates its context again, and repeats the cycle until the response is complete.
For a sufficiently large model, every iteration requires moving enormous quantities of model data through GPU memory. During low-concurrency inference, the GPU can often execute the necessary arithmetic faster than its memory subsystem can deliver the data required for the next operation.
The result is counterintuitive: some of the world’s most powerful processors can spend significant portions of LLM generation waiting on memory rather than exhausting their computational capacity.
That bottleneck helps explain why a model running on extremely expensive hardware can still appear to type its answer only a few dozen tokens per second.
It also explains why one of the most important LLM inference optimizations does not attempt to make the model smarter, smaller, or less accurate.
Instead, it changes how many useful tokens the system can obtain from each expensive pass through the large model.
The technique is called speculative decoding, and under the right workloads it can make generation roughly two to three times faster without replacing the target model with a weaker one. The original speculative-decoding research demonstrated 2×–3× acceleration while preserving the target model’s output distribution.
For AI infrastructure teams, platform engineers, product leaders and organizations operating increasingly large fleets of inference GPUs, this is more than a latency trick. It points toward a broader shift in AI infrastructure: competitive advantage is increasingly coming from using existing compute more efficiently rather than simply adding more compute.
Where LLM Latency Actually Comes From
It is tempting to think of AI inference as one continuous operation, but a request to a language model actually contains two very different performance problems.
The first is prefill.
When a user submits a prompt, the model initially processes all of the input tokens. It converts the prompt into the internal representations needed for generation and populates the model’s key-value cache, or KV cache, which stores information that can be reused as generation progresses.
This stage can perform substantial amounts of work in parallel. Large matrix operations give the GPU enough computation to use its tensor cores efficiently, making prefill relatively compute-intensive.
A longer prompt generally means more prefill work. That is why systems processing enormous documents or very long conversation histories can take noticeable time before displaying anything at all.
The user-visible metric associated with this phase is usually Time to First Token, or TTFT.
Then the model enters the second phase: decode.
During decoding, the model generates the response one token at a time. The next token depends on everything generated before it, which means the process cannot simply calculate an entire answer in parallel.
Generate token one.
Update the sequence.
Generate token two.
Update the sequence.
Generate token three.
And continue.
That sequential dependency creates a very different hardware workload. vLLM’s optimization documentation characterizes prefill as compute-bound and decode as memory-bound, which is why techniques such as chunked prefill attempt to combine the two types of work to improve GPU utilization.
For users, decode performance determines inter-token latency: how quickly one visible token follows another once the response begins.
This distinction is important because two AI applications can both be described as “slow” while suffering from completely different infrastructure problems.
An enterprise search system feeding tens of thousands of tokens into a model may primarily have a TTFT problem. A coding assistant that starts responding quickly but takes twenty seconds to produce a long function may primarily have a decoding problem. A voice assistant needs both phases to be extremely fast because delays of even hundreds of milliseconds can disrupt the conversational experience.
Speculative decoding is primarily designed to attack that second bottleneck.
 |
Why Generating One Token Can Require Moving So Much Data
Consider a dense model containing 70 billion parameters.
If those parameters are stored using BF16 or FP16 precision, each parameter requires approximately two bytes. The model’s raw weights therefore occupy roughly:
70 billion parameters × 2 bytes ≈ 140 GB.
That does not mean every real-world deployment literally transfers exactly 140 GB through one GPU every time the user sees a token. Large models can be quantized, partitioned across several accelerators, distributed through tensor parallelism, or implemented using architectures such as mixture-of-experts where only part of the model is active.
But the number demonstrates the fundamental scale of the memory problem.
Those weights are far too large to remain inside the GPU’s small high-speed caches. During decoding, the accelerator repeatedly needs access to large portions of model parameters stored in high-bandwidth memory.
Now compare that requirement with the memory bandwidth available from modern hardware.
NVIDIA lists approximately 3.35 TB/s of memory bandwidth for the H100 SXM and 4.8 TB/s for the H200 SXM, with the H200 providing 141 GB of HBM3e memory.
A simplist