|
Agents Can Now Sign Up for Your App (Sponsored)
 |
Agents are hitting your signup flow and bouncing off a browser login built for humans. Every one that gives up is a signup you never see.
WorkOS Agent Registration turns that traffic into signups. Enroll via the dashboard and AuthKit publishes an auth.md file agents read to register for scoped, short-lived credentials you control.
What does it take for a social media platform to stop rewarding clickbait content?
At first glance, this question might sound like a moderation problem that you can simply solve by enforcing better content policies and classifying posts that are engagement bait. However, the problem is much deeper. It often sits inside the component that decides which posts become part of a user’s feed in the first place.
Consider the scale of the decision. When you open a feed, the platform has a few hundred milliseconds to select a handful of posts from hundreds of millions of candidates. Scoring every candidate with an expensive model burns through the time budget. To get over this, the social media platforms rely on engagement, which is a cheap and somewhat reliable proxy for judging relevance. Such a proxy is easy to measure and optimize against, and it powered a generation of recommendation systems.
 |
However, the problem is that this type of engagement is also easy to manufacture. For example, a post that opens with “comment DONE if you’re a real engineer” collects clicks and replies while delivering little value. But a ranking function tuned to reward interaction will promote it even though it is clearly an engagement bait. For years, the countermeasures against this were heuristics and demotions applied after ranking, but such an approach hasn’t eliminated the problem. Accounts producing bait always find ways around such measures.
Over the past two years, LinkedIn, Meta, and YouTube, three of the largest platforms, have tried to address the root of the problem. All of them have attempted to rebuild the retrieval stage around the meaning of content, matching posts to people by what a post is about and how it relates to a reader’s interests. The idea is that once the relevance depends on semantic meaning, the tactics built for engagement farming lose their potency. However, the three companies have taken three different directions to solve the same problem, which provides us with an opportunity to understand things from multiple perspectives.
In this article, we will work through the following points:
Why feeds relied on engagement signals for so long, and where that approach reached its limits.
How embeddings match users and content by meaning.
LinkedIn’s consolidation of five retrieval systems into a single language-model retriever.
Meta’s opposite choice of keeping a large family of specialized models arranged as a funnel.
YouTube’s generative approach, where the system produces the identifier of the next item.
The cold-start problem, and why pretraining helps most when a user’s history is thin.
The tradeoffs each design carries, and the limits of the engagement-bait result.
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
Engagement Signals
Every large feed runs on a two-step pipeline:
Retrieval: This step reduces hundreds of millions of candidates down to roughly a thousand. It has to be cheap because it touches the entire corpus.
Ranking: This step spends real compute ordering those survivors into the sequence you scroll.
Most of the recent architectural changes sit in retrieval, so that is where we will focus more.
For years, retrieval relied on behavioral signals. The system recorded which posts a user clicked, watched, and reacted to, then retrieved content that resembled those interactions or resembled the behavior of similar users. This approach scales well and produces reasonable feeds.
However, it also has a weakness.
When a system optimizes a single measurable objective, it tends to optimize the literal metric rather than the intent behind it. As mentioned, engagement is a proxy for relevance, and a proxy can be optimized directly. A retrieval stage tuned on interaction counts will surface whatever maximizes interaction counts, and engagement bait is the content that does exactly that. For example, content designed to trigger a click or a reply scores highly on the measured signal while contributing little to the experience the platform aims to deliver.
Suppressing such content through demotions and rules tries to treat the symptom without fixing the underlying disease. The retrieval stage still keeps bringing up the same material. A more durable fix changes what retrieval measures in the first place, and this happens by moving from behavior to meaning.
Semantic Retrieval
The alternative is to retrieve content by its semantic content rather than by its interaction history. The ability to make this possibility depends on embeddings.
An embedding is a list of numbers that positions an item as a point in a high-dimensional space, arranged so that related items land near one another. For example, a post about fixing a leaking tap and a post about reducing water waste sit close together even when they share no keywords, because their meanings are related. In more technical terms, the embedding reflects the relationship captured during model training.
To use embeddings for retrieval, platforms apply a dual-encoder design, sometimes called a two-tower model.
One encoder converts a user, along with their profile and recent activity, into a point in the space.
A second encoder converts each post into a point in the same space.
Since the two encoders operate independently, a plat