OpenAI’s New Prompt Caching Changes the Economics of Long-Running AI Agents

The Network Layer Problem Most Privacy Stacks Don’t Address (Sponsor)

Your carrier operates at a layer your privacy settings can’t reach.

Privacy engineering lives at the application layer — encryption in transit, zero-knowledge design, data minimization. The network layer gets less scrutiny.

Your carrier sits below all that. It sees your IMSI — a static identifier tied to your SIM that persists across every tower connection, regardless of VPN, DNS, or OS. It also sees your real number, call and text metadata, and location via tower triangulation. No application-layer config touches it.

Cape rotates the IMSI every 24 hours, resetting the trail at the only layer where the problem lives.

How IMSI rotation works

The expensive part of an AI agent is not always generating the answer. Increasingly, it is repeatedly reading everything the agent already knows.

A long-running coding agent might carry tens or hundreds of thousands of tokens containing system instructions, repository context, tool definitions, previous edits, test results, terminal output, architectural documentation, and conversation history.

Then the user asks something surprisingly small:

“Run the tests again.”

Without effective caching, much of that enormous context may need to pass through the model again before the model can respond.

OpenAI’s latest improvement to prompt caching for GPT-6 attacks exactly this problem. The company says GPT-6 now delivers higher cache-hit rates by default, while adding explicit cache breakpoints, diagnostics, cache prewarming, and mechanisms for changing tools or reasoning effort without unnecessarily invalidating cached context. Cached input-token reads receive discounts of up to 90%.

On the surface, this sounds like an inference optimization. For agent architecture, it is something larger.

It means developers increasingly need to think about an LLM prompt the same way systems engineers think about memory hierarchies, database indexes, CDN caches, and incremental computation.

The question is no longer merely:

What context should the model receive?

It is also:

Which parts of that context can remain structurally stable enough that the model never has to compute them twice?

The Hidden Cost of Persistent AI Agents

Traditional chatbot interactions are relatively straightforward.

• A user submits a prompt.

• The model processes it.

• The model generates an answer.

• An agent works differently.

Consider a coding agent operating inside a large repository. Its first request might contain:

• developer instructions,

• repository conventions,

• security constraints,

• dozens of tool definitions,

• architecture documentation,

• relevant source files,

• conversation history.

The agent calls a tool.

The tool returns output.

Another model request happens.

The agent edits a file.

Another request.

Tests run.

Another request.

The developer asks a follow-up.

Another request.

The important point is that most of the context may be identical between these calls.

A session could therefore contain a large stable prefix followed by a relatively small changing suffix.

Historically, that creates an unfortunate scaling property: the longer the agent works, the more context it accumulates, and the more expensive later turns can become.

Prompt caching changes that relationship.

Instead of recomputing the unchanged portion every time, OpenAI can preserve the model’s previously calculated state and reuse it.

What Is Actually Being Cached?

Transformer models generate internal key-value states, usually called the KV cache, while processing tokens.

These states represent information the attention mechanism needs when later tokens refer back to earlier parts of the sequence.

With prompt caching, OpenAI preserves those KV states for reusable prompt prefixes. If a subsequent request begins with exactly the same cached prefix, the model can reuse those calculations instead of recomputing them. OpenAI’s documentation emphasizes that the cache stores the resulting KV tensors rather than simply storing the prompt tokens themselves.

Conceptually: