Applied AI

Should businesses worry about whether leading AI providers are using their corporate data to develop competing products? The AI providers’ rivals, including the CEOs of Microsoft, Salesforce and Palantir, seem to think so

The real threat is far more subtle, having more to do with the high-level usage data AI providers can collect and a loophole that involves customers voluntarily sharing their chats with chatbot creators.

Traditional software CEOs are pushing their own products to help businesses use the most advanced AI without sharing their information with the AI model creators in the first place.

Microsoft CEO Satya Nadella warned in a post Sunday that providers of AI models are gleaning valuable information from their customers to eventually compete with them.

“In the AI age, the buyer risks giving away knowledge, just in order to use what they bought,” he wrote on X. “It’s the kind of knowledge a competitor could never buy, and the kind that leaks almost imperceptibly.”

Nadella’s post cited Palantir CEO Alex Karp, who said earlier this month that customers should worry that firms like Anthropic and OpenAI are “gonna take the alpha of my business, transfer in [such data to] their [AI model] weights, and compete against me.”

Later Sunday, Salesforce CEO Marc Benioff published a similar post stating that his company, which uses models from Anthropic and OpenAI, had “drawn a hard line” by refusing to let them train on Salesforce customers’ data.

Anthropic and OpenAI both say they don’t use enterprise customers’ prompts or responses—essentially the chat logs between those firms and Claude or ChatGPT—to improve their models. But both companies say they improve their models using chat logs from some individual users depending on which privacy settings those users choose, and many businesses worry their workers are incidentally giving away proprietary data when they use the chatbots for work.

Are there other ways Anthropic or OpenAI could be using information gleaned from customers to their own advantage?

Perhaps. Anthropic launched Claude Code last year after AI coding app Cursor had used Claude models to drive booming traffic for coding tasks. More recently, Anthropic blindsided some business partners that had used Claude to power apps for designers or lawyers when it later launched its own, competing apps or features for people in those fields.

This is nothing new. Similar concerns have long existed around the owners of any major technology “platform,” such as an operating system, search engine, social network, ecommerce marketplace or cloud server provider. Microsoft, Google, Facebook and Amazon have all developed products that competed with businesses that previously launched applications or products on their respective platforms. This phenomenon is also known as “platform risk.”

As for concerns involving more granular data, cloud providers like Microsoft and Amazon in some cases can give their customers ways to access AI models without sending their chat logs back to the AI providers.

For instance, Microsoft offers “dedicated instances” of OpenAI’s models that let companies use the models on a dedicated set of cloud servers without sending any data, such as their chat logs, back to OpenAI; Amazon offers a similar service for Anthropic’s models. This type of arrangement is currently the norm among most large enterprises, especially financial institutions and sophisticated tech firms that are paranoid about data privacy, we’re told.

Nadella’s argument is self-serving, of course, as Microsoft itself doesn’t have an advanced AI model that competes with those from Anthropic. Nadella needs to remind customers of Microsoft’s relevance in a future where such frontier models are used widely. Palantir and Salesforce are in similar positions; Anthropic and OpenAI also threaten these companies’ core enterprise application businesses.

But that doesn’t diminish the concerns enterprises may have about handing over sensitive data to AI models. We’ve heard from some companies who have declined to use cutting-edge models from Anthropic due to the startup’s insistence on temporarily holding onto their chat logs, which Anthropic said was necessary for security-related reasons.

Anthropic says enterprise customers can choose to provide feedback on Claude by clicking a thumb’s up or down button and filling out a text box, and in those cases Anthropic can use chats and coding sessions as training data for its models.

Elsewhere, some customers have noticed that Claude has appeared to edit their chat logs based on information like their location and the tools they use, raising the question of whether it’s retaining data about how customers use it. (An Anthropic spokesperson said that the practice was an “experimental anti-abuse measure” aimed at detecting whether people were using models for banned purposes, such as Chinese AI developers trying to use Claude to train their own AI models.)

Other leading American AI labs have also raised eyebrows over how they treat customer data. SpaceXAI drew criticism from some developer customers over the weekend after they noticed the company was storing code they uploaded to its Grok AI coding tool if they failed to opt out of such retention. CEO Elon Musk appeared to backtrack on the policy on Monday, stating that “as a precautionary measure, all user data that was uploaded to SpaceXAI before now will be completely and utterly deleted.”