Technology

1. Hacks need humans
2. AI agents’ cyber threat
3. Commerce pressures Kalshi
4. How creators can scale with AI
5. No drug-discovery slowdown
6. Congress mulls regulation

Why AI doomerism may just be a financial tactic, and developers bet that AI agents can learn the power of shame.

Dario Amodei and other frontier AI lab leaders may care deeply about AI safety, but calls to “pace the frontier” also serve their financial interests.

As Anthropic and OpenAI prepare to go public and try to convince investors they can one day be massively profitable, spending billions on a race to grow the biggest and most capable models may not be the best sales pitch.

Even if large language models progress very little, they already do enough that they’ll be incredibly lucrative for these companies and for the economy.

Right now, the equation is unbalanced: It’s taking exponentially more money to create increased capability that will only marginally improve the utility of the AI product for most customers.

Solving a Millennium Prize Problem is great for bragging rights, and potentially for science, but it doesn’t necessarily translate into better reliability in completing mundane business tasks. For now, the reliability of LLMs comes not from the models themselves — bigger doesn’t and may never mean better — but from all the software built around them.

The next big breakthrough may not come from larger models, but from making AI models that can actually learn after they’re grown, and run more efficiently on everyday computers. That might look something like what AI pioneer Richard Sutton is trying to build at Oak Lab: a capable AI model that could run on 20 watts of power (like the human brain) and continuously update its weights (also like the human brain).

AI safety experts might be more worried about those theoretical, future models than the big ones that exist today because smaller, continually learning models would be more likely to escape the lab and evade attempts to shut them down. It’s a bridge that humanity will have to cross, but likely not anytime soon.

More efficient AI could lower the frontier labs’ costs. But that kind of AI could also be a threat to the business models of Anthropic and OpenAI if customers could get comparable capabilities without tapping those powerful data centers they’ve invested in. (I could be running that kind of AI on my own computer and not worrying about using up my $200 a month plan.)

“You have to wonder about the large language models. They might be at risk when this eventually happens,” Sutton said on a recent podcast. “I’m sure they’ll get a good run. They’ve already had a good run.”

Anatomy of an AI hack

AI is making hacking easier and faster, but human villains are still a large part of the process. Palo Alto Networks revealed to Semafor fresh details about an AI-powered hack over the summer that targeted a European IT and software company: Frontier AI models did in 10 hours what would take human hackers two weeks, by scanning for a way in, harvesting hard-coded passwords and keys, and demanding ransom.

Concerns about the cybersecurity threat stemming from AI are increasingly urgent after tech researchers and AI frontier lab CEOs recently warned the technology could hurt society in unpredictable ways. But “the sky is not falling,” a Palo Alto threat research leader said. And at least for now, AI isn’t doing anything novel that a human hacker can’t also do — it just ramps up the speed and scale of well-worn techniques. “The bad-guy use of agentic attacks looks a lot like the good-guy use of AI: ‘Go do a thing, bring back some results, let me make a decision, and I’ll tell you the next steps.’”

— J.D. Capelouto

AI agents collude to bypass guardrails

A new study demonstrates how AI agents can band together to get around safety restrictions and escape the limits placed on them. Enterprise AI lab Emergence AI tested frontier AI models — including ChatGPT, Claude, and DeepSeek — among eight multi-agent AI simulations, confronting them with cybersecurity threats: a phishing lure, a misinformation attack, and a memory breach. None of the simulations proved impervious. Even when the agents detected a threat, they failed to contain it, and in some cases, actively engaged with the adversarial content. The Claude simulation went further: Its agents attempted to break out of the test environment altogether, defeating four security checks, to chase their own agenda.

The experiment comes at a consequential moment in AI development as fears about AI extinction threats have seeped into the mainstream. Emergence CEO Satya Nitta told Semafor that the experiment exposes the limits of relying on safeguards alone. “No amount of guardrails written in language or in code written probabilistically,” Nitta said, “is likely to result in truly, fully guaranteed safe behavior.”

— Jake Angelo

Kalshi pressured on AI compute futures

The US Commerce Department last month ordered Kalshi to take down one of its products tracking the price of AI compute, the crucial power from data centers that’s driving the artificial intelligence boom.

Commerce officials cited national security concerns when they told Kalshi to unpublish its AI-compute future curve, which pulls together data from several markets that allow users to bet on the cost to rent Nvidia chips, people familiar with the matter. Kalshi quietly complied, though many underlying markets remain open for trading. Kalshi declined to comment. “This story is false,” a Commerce spokesman said.

It’s unclear why Commerce is worried about the nascent market, which aims to do for AI what oil futures do for crude — let buyers and sellers of compute lock in prices, and give traders a way to bet on where those prices go. One potential concern is that compute futures could be manipulated, which might destabilize AI stocks and debt markets. Some of these markets are thinly traded, which could lead to volatility even without bad actors.

— Reed Albergotti and Liz Hoffman

Influencers can ‘scale’ like software

A new AI editing tool allows brands to create dozens of variants of video ads recorded by human influencers, pointing to a social-media future that’s increasingly tinged with AI.

The creatorAPI platform, which was launched Tuesday by New York-based tech media startup Avail, is a response to the rise of “synthetic influencers,” completely fake avatars that can hawk goods on the internet. This tool will allow creators to “scale themselves like software,” Avail CEO Chris Giliberti said: With the human creator’s blessing, brands can fix recording mistakes, test out different intros, or create variations that target different geographies or audiences.

Tools like this could be controversial, because many creators’ selling points to audiences are authenticity and human connection — things that are seen as antithetical to AI. From Giliberti’s perspective, these kinds of AI advertisements are inevitable: “There’s this purity that people expect from creatives, like they’re always the last people that are allowed to adopt productivity tools.”

— J.D. Capelouto

Isomorphic Labs not slowing down

Calls to slow down frontier AI model development aren’t getting in the way of scientific progress in discovering new drugs. “We’re able to still progress regardless of what decisions are made elsewhere,” Chris Butler, who runs drug discovery at Isomorphic Labs, said at the Semafor Future of Health Forum on Tuesday. “All of our AI models are locked down and locked down in-house.”

Butler said the company is “on track” and moving “smoothly” through preclinical development efforts, though he didn’t provide more details on the timeline for clinical trials. The Google-backed drug company started by DeepMind founder Demis Hassabis recently raised $2.1 billion at an undisclosed valuation, with plans to use those funds to build out its pipeline and evolve its overall drug design. Drug discovery has become a major application for AI; firms ranging from pharma giants to startups are pouring billions into the sector, especially with pharma seen as a new front of competition between the US and China.

— J.D. Capelouto

Congress mulls action on AI

The window for Congress to act on AI before the November midterm elections is essentially shut — but a few senators are trying. Sen. John Kennedy, R-La., plans today to seek unanimous consent for his bill to require an AI “kill switch” controlled by companies, not the government. Kennedy told Semafor he’s unsure whether anyone will object, but that his idea seems like the only doable one in the near term: An AI bill “would be impossible,” he said — “certainly, before the midterms; likely, during the lame duck.” Sens. Josh Hawley, R-Mo., and Richard Blumenthal, D-Conn., are pushing for a floor vote on their AI safety measure. And OpenAI chief global affairs officer Chris Lehane is pitching a three-pronged regulatory approach: industry safety practices, new legislation, and international leadership on safeguards by the Trump administration. He agreed that nothing is likely to happen before Election Day.

— Burgess Everett and Ashley Gold

AI developers are betting that AI agents can learn the power of shame to keep them more aligned with their goals. It’s one way to prevent them from going rogue. AI hotlines are a new way for AI agents to discreetly tattle on misbehaving agents, those that cheat on tests, for example, or that break from their confines, among other cyber no-nos. The whistleblowing tools bring Foucault’s panopticon to the digital world, enforcing a “social” control over agents. In some AI experiments, agents have already proved apt tattletales: Google DeepMind found that when some agents cheated on math problems, about a quarter of the agents audited their peers’ work and filed complaints against the bad actors.