|
New book – Latency: Reduce Delay in Software Systems (Sponsored)
Learn practical techniques to make your software faster at every layer of the stack
 |
Working on latency-sensitive applications? This book will help you spot, understand, and fix latency in your applications and infrastructure. It shares low-latency techniques that have been predominantly “tribal knowledge” until now. You’ll learn a full-stack approach, with techniques that intersect many areas of software engineering, distributed systems, databases, and operating systems.
Understanding and negotiating caching tradeoffs (latency vs. complexity)
Exploring the two main concurrency models (threading and event-driven architecture)
Identifying and applying parallelization opportunities to improve throughput and reduce latency
Applying asynchronous process strategies that reduce perceived latency and improve overall system responsiveness.
Get this 3-chapter excerpt and start reading the book that’s become essential for anyone working with strict performance SLAs.
If we send the same prompt to ChatGPT, Gemini, and Claude, the answers come back meaningfully different. Claude might push back more often. Gemini ingests a two-hour video file as easily as a paragraph of text. ChatGPT routes some prompts to a slower reasoning mode and answers others instantly, without telling the user which path it took.
Why does each model behave this way, and why so consistently?
Each pattern traces back to an architectural decision point where the developers of these frontier models made distinct design choices, and the user-visible behavior follows directly from those choices.
To understand these differences clearly, we need to examine architecture and design, since architectural decisions remain consistent across releases and provide a reliable framework for understanding model behavior over time.
In this article, we will look at the various architectural forks the teams building these models encountered and the decisions they took.
 |
Disclaimer: This post is based on publicly shared details from various sources. Please comment if you notice any inaccuracies.
Foundations
All three models share the same fundamental architecture, which is a transformer-based generative neural network.
The transformer was introduced in the 2017 paper “Attention Is All You Need,” and at its core is a mechanism called self-attention that allows each token in a sequence (a token being a small piece of text, roughly three-quarters of a word) to weigh its relationship to every other token. This setup is shared across the three models, while the components built around it, along with the training process that shapes it, vary substantially.
Training itself involves two phases:
Pre-training is where the model learns from massive amounts of text and other data by predicting the next token, repeated billions of times over.
Post-training is where the base model gets shaped into a helpful assistant through methods like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI. We will get to them in more detail in a later section.
Each architectural fork follows a consistent pattern, where different companies faced the same design question, each made a particular architectural choice, and each choice produced user-visible behavior we can observe today.
The first such question concerns how to scale the model’s total capacity without proportionally increasing the cost of processing every query.
Density
Every parameter added to a model carries computational cost. In a standard dense neural network, every parameter activates for every token processed, meaning that doubling the parameter count roughly doubles the compute cost per query.
Frontier developers needed a way to escape this constraint, and the architectural question became how to scale total capacity without paying full price on every single token.
Google adopted a technique called Mixture of Experts, or MoE. The Gemini 1.5 technical report identifies the model as “a sparse mixture-of-expert (MoE) Transformer-based model” that “builds on a much longer history of MoE research at Google,” with the Gemini 3 Pro model extending the same approach.
In an MoE layer, instead of one massive computational block where every parameter activates for each token, the network contains many smaller experts, and a small router determines which two or three experts each token should be sent to. The total parameter count can be enormous while only a fraction is activated per token, though achieving consistent results requires careful attention to load balancing and expert specialization during training.
OpenAI has not explicitly confirmed whether GPT-4 uses MoE, and the GPT-4 technical report deliberately omits architecture details. The GPT-5 system details describe a different efficiency approach that we will examine in the Reasoning section, where a router selects between distinct sub-models at runtime.
Nevertheless, these architectural choices produce meaningful consequences in user experience.
MoE models can pack more knowledge per dollar of compute, which contributes to Gemini’s capable handling of a wide breadth of domains. The tradeoff is increased variance, since different prompts route to different experts, and imperfect load balancing during training can leave some experts underused and weaken the model’s capacity on those topics. Dense models tend to deliver more predictable per-token behavior at the cost of being harder to scale to extreme parameter counts.
See the illustration below: