|
What is loop engineering? (Sponsored)
 |
Every agent already runs a loop. Loop engineering adds a loop around the agent itself, enabling it to evaluate its output, try again when the work falls short, and refine its instructions when the same mistakes recur. Today, you perform that role: reviewing the work, diagnosing what went wrong, and prompting the agent again. This article shows how to automate that process with a working example, while exploring where human judgment still belongs.
A 70-billion-parameter model requires reading roughly 140 GBs of weights out of the GPU memory. On a modern data center GPU, this transfer can take tens of milliseconds. The actual calculation applied to these weights takes a fraction of that time. This means that the processor’s math units are unused for most of the time taken by the token generation step.
Speculative decoding is a technique that converts this unused capacity into output. A second, much smaller model produces several candidate tokens in advance. The large model evaluates all of them in a single forward pass instead of one pass per token, resulting in 2-3 times faster generation. To make things better, the text produced remains statistically identical to the output of the large model running alone.
In this article, we will look at how speculative decoding works. Here’s what we will cover:
Why token generation runs one step at a time
What a GPU spends its time on during generation
How several candidate tokens are evaluated in a single pass
The accept and reject loop, and what happens when a candidate is wrong
Why output quality is preserved exactly
Acceptance rate, and why it varies by workload
The four places a draft can come from
When speculative decoding stops helping
 |
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Autoregressive Decoding
Text generation works one token at a time.
The model reads everything produced so far, computes a probability distribution over its vocabulary, selects the next token, appends that token to the input, and repeats the cycle. Each cycle is called a forward pass, and every forward pass runs the input through all layers of the model.
For example, token 50 depends on token 49 being present in the input, and token 49 depends on token 48, and so on. Computing them simultaneously would break the dependency chain that makes the output coherent.
The implication is that a 500-token response requires 500 sequential forward passes, each one completing before the next begins. Since the duration of a single pass depends on the size of the model, the total generation time equals the number of output tokens multiplied by the time per forward pass.
This explains why the response speed stays roughly steady whether the answer is a short factual reply or a long block of code, because the per-token cost stays the same either way. It also explains why a larger model produces text more slowly on identical hardware.
Modern inference systems use a KV cache, which stores the attention state for tokens already processed so that each new pass only computes attention for the newest position. This cuts the work done inside each pass to a large extent, though the requirement for one pass per token still remains.
Memory Bandwidth
Since the number of passes is fixed by how much text we want, it leaves the second half of the equation. What does a single forward pass actually spend its time doing?
To put it simply, a forward pass spends most of its duration moving data rather than performing arithmetic calculations.
Model weights live in the GPU memory, usually called VRAM. To compute anything with those weights, the GPU has to transfer them into the compute units where the multiplication happens. For a 70-billion-parameter model stored at 16-bit precision, this transfer amounts to roughly 140 GBs for every single token.
The arithmetic performed on those 140 GBs is quite small by comparison. One token means one narrow vector flowing through each weight matrix. The GPU loads an enormous matrix out of memory, multiplies it against that vector, discards it, and loads the next one.
The consequence is that during prompt processing, compute utilization is around 90 to 95 percent. However, during token generation, it falls to somewhere between 20 and 40 percent. The math units are unused for most of every step while the memory bus runs near capacity.
The difference is driven by how much work each weight read supports:
Prompt processing reads the weights once and applies them to thousands of input tokens simultaneously.
Token generation reads the same weights and applies them to exactly one token.
This is capacity that has already been paid for, but underutilized.
But why does this matter practically?
A GPU with higher memory bandwidth improves generation speed more than one with more raw compute.