|
Optimizing Write-Intensive Database Performance (Sponsored)
Free masterclass: Learn practical strategies for predictable low-latency writes at scale
 |
Free Masterclass: Optimizing Write-Intensive Database Performance
Database writes – at scale – are one of the hardest problems in distributed systems. This masterclass will teach you how to understand and avoid latency spikes in real-time, write-heavy database workloads. Our panel of experts will share a practical framework for diagnosing write bottlenecks and knowing which strategies to apply in different scenarios.
After this free 2-hour masterclass designed for developers, engineers, architects, and database practitioners, you will know how to:
Identify which factors (database internals, database configurations, data modeling…) are impacting your write performance
Avoid mistakes that have caused 40x write amplification in production
Sustain low P99 latencies even during sustained growth and volatile spikes
All attendees will get the complete Database Performance at Scale book by the masterclass instructor Felipe Mendes.
How does a website charge a visitor who arrives anonymously, skips every advertisement, and leaves within a second?
For most of the web’s history, the question rarely came up, because the visitor was a person whose attention a site could sell through an ad or a subscription. If the website had good content, the website owner could count on multiple such visits by the same user. But now the visitor being a person is not always true. More than half of the traffic online now comes from software that acts on a person’s behalf, requests a page, and leaves without engaging with any ads or subscriptions [2].
Cloudflare seeks to change this.
As you might be aware, Cloudflare sits between a large share of the world’s websites and everything requesting them. It works as a reverse proxy that each request passes through before it reaches the origin server [1]. This lets Cloudflare read a request and act on it early. Over the past year, the company has used this position to sort automated traffic by what it does, to verify the identity behind a request, and, most recently, to collect payment for a request through an open protocol named x402 [1].
In this article, we will go through Cloudflare’s solution in the following five steps:
The web’s usual way of earning value leans on human attention, and agent traffic changes it fundamentally.
A look at Cloudflare’s initial solution around blocking automated traffic and then charging for each crawl.
Settling identity, permission, and payment inside a single request.
A look at how x402 exchange completes this settlement.
The x402 exchange completes this settlement through a short back-and-forth over ordinary HTTP
The costs and open questions of this approach
 |
Disclaimer: This post is based on publicly shared details from Cloudflare. References at the end. Please comment if you notice any inaccuracies.
The Attention Model
For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit [1]. The request itself stayed free, and this setup funded a large part of the Internet.
A growth in agent traffic is changing this dramatically. An agent is software that acts on a person’s behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention [1]. Each of the three old settlement points depended on a person staying long enough to be counted, so software traffic leaves those points idle.
The strain here is that this kind of traffic is now fast becoming the majority. More than half of the requests reaching websites come from software rather than people [2]. This means that request volume climbs while revenue stays flat.
To make things clear, parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key [1]. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned [1].
If the value used to settle downstream of the request, the natural question is which party is positioned to move that settlement back onto the request. The answer starts with where Cloudflare sits.
The Proxy Layer
A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare’s network first and passes through it on the way to the origin [1]. The origin is the site’s own server, the machine that ultimately holds the page or runs the API.
On a side note, the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin. This broader capability is quite consequential at Cloudflare’s scale. [2].