|
Only Pay for Fine-Tuning That Works (Sponsored)
 |
GPU-hour billing charges for the entire time a machine is reserved — setup, idle time, queueing, and failures — so your cost depends on infrastructure efficiency you don’t control. Crusoe Serverless Fine-Tuning bills per token processed during training, making costs predictable from your dataset size and epoch count, with no GPUs to provision or right-size. Early stopping ends the job, and the billing, the moment your model stops improving — you’re never charged for epochs that add no accuracy. Jobs run on AI-optimized infrastructure with automatic recovery and restart, and checkpoints save at every step.
AI labs are moving faster than ever and releasing the most capable models we have ever seen. Just recently, Anthropic released Fable 5, then OpenAI released the GPT-5.6 family, including GPT-5.6 Sol, their most capable model yet, Kimi released Kimi K3, and Opus 5 came out just a few days ago.
But capability is only half of the picture. The other half is how much it costs these models to complete tasks, that is, the cost per successful task. Lower cost makes the model more affordable for users and less costly for the provider. A huge amount of effort inside the labs goes into making every component and layer of their AI applications optimized and more efficient, to reduce the overall cost. For example, GPT 5.6 Sol with max reasoning scores higher than Fable 5 on the Artificial Analysis Coding Agent Index while costing less than half as much.
To understand what techniques are adopted in frontier labs to make AI applications more efficient, we met with the OpenAI engineers who developed and shipped various efficiency techniques into the systems behind Codex and ChatGPT Work. Thanks to Joe, Ahmed, Steve, Matthew, and Philippe for sharing valuable details with us.
In this article, you’ll learn:
What actually happens when you send a request to an AI agent
How the harness layer cuts repeated work with persistent WebSockets, stable prompt prefixes, deferred tool discovery, and Code Mode
How the API layer tokenizes only the delta and runs safety checks in parallel with inference
How the inference layer gets more out of GPUs with cache-aware routing, KV cache management, speculative decoding, and separating prefill from decode
What OpenAI learned from all of this, and which lessons you can apply to your own systems
 |
Anatomy of an Agentic AI Application
When you give Codex or ChatGPT Work a task, like fixing a bug, the query does not go directly to the LLM. It is more complex than that. An AI application like Codex is not just an LLM. It is a system, with many components. User queries pass through several layers before the LLM ever sees a single token.
 |
But why can’t we send the user query directly to the LLM? To understand why, let’s see what an LLM actually is.
An LLM is a neural network trained to predict the next token. It takes a sequence of tokens as input and produces a sequence of tokens as output. It cannot run a shell command, edit a file, or remember anything between calls. But an agent task like “fix this bug and run the tests” is mostly actions. Something has to turn the model’s predicted tokens into real commands, feed the results back, and continue until the task is done.
That is the job of the harness layer, a system built on top of the LLM to handle those responsibilities. It takes the user’s task as input, decides which instructions, which tool definitions, and how much history to include in the context, and maintains the conversation history. When the model responds with a tool call, the harness executes it under approval policies in a sandbox environment, appends the result, and sends the conversation back to the LLM.
 |