Could your chatbot tip off a scientific rival? It’s possible, researchers say

From ScienceMag:

A familiar pattern is beginning to emerge when artificial intelligence (AI) makes a scientific breakthrough. After an AI company announces the feat, a researcher who uses that company’s tool claims it may have ripped off their work.

On 8 September, for example, OpenAI said its agents had solved the Navier-Stokes problem, one of the biggest open challenges in math. But mathematician Tristan Buckmaster of New York University publicly suggested the company had benefited from extensive work he and a colleague, Levent Alpöge, had done using OpenAI’s coding assistant, Codex.

Two weeks later, Anthropic said its model, Claude, had found a new enzyme system. Mario Rodríguez Mestre, a computational biologist at the University of Copenhagen, soon told The New York Times his group had been studying the same systems since 2022, and had shared unpublished drafts with Claude.

OpenAI says an investigation confirmed Buckmaster’s Codex work from the prior 2 months could not “have influenced the model or the math results it produced in any way.” Anthropic says Claude was “not trained on any user transcripts.” Neither researcher says they can prove otherwise.

Still, scientists are now asking an uncomfortable question about tools many use daily. When you type an unpublished idea into a chatbot or coding agent, could it end up being used by the company that built the model—or a rival who asks the right question a year later? And what can you do to reduce the risk?

Many AI companies give researchers—who are also potential customers—free or reduced-price access to their tools, and some worry there is a quid pro quo. “People working in a university environment are a much more valuable segment of the market,” says Mohammad Hosseini, who studies the ethical implications of using AI in research at Northwestern University. “And perhaps their information is used more frequently to train these models.”

In principle, an unpublished proof or paper could become a part of a model’s next training run, he and others say. But can such a tiny drop in the vast ocean of training material still make a splash? There’s some evidence it can. For example, a preprint published in January showed chatbots could reproduce most of a Harry Potter novel nearly word for word.

But such memorization isn’t really the threat for research, says A. Feder Cooper, an AI researcher at Yale University and co-author of the study. “The data sets that these models are trained on are absolutely massive … too large from an information theoretic standpoint for the models to memorize all of it,” Cooper says. But he adds that an AI model doesn’t need to memorize a chat word for word for it to be useful. “It just needs to get enough of an understanding from the chat … to use that information.”

Indeed, a single user’s chats may not matter at all. What may matter more is how thousands of chats add up. AI companies routinely analyze conversations in bulk to see how people use their products. Anthropic, for example, has described a system called Anthropic Insights that summarizes millions of chats and sorts them into clusters. To protect privacy, it discards clusters with small numbers of users, but it doesn’t say what that threshold is.

Such analyses are the likeliest way sensitive research information could leak, says AI privacy researcher Niloofar Mireshghallah of Carnegie Mellon University. The risk isn’t copying, but a company noticing from aggregate chats that a problem is close to being solved and putting its resources behind it. “That gives them enough hints of: ‘Oh, this is solvable, let’s go do it.’”

The privacy safeguards on these analyses may also fail researchers, Mireshghallah adds. They protect users by hiding them among thousands of others with similar chats, but a small number of researchers sending many queries about the same niche problem can still stand out. “Average case privacy methods don’t protect the outliers,” Mireshghallah says.

No setting makes a chatbot fully private, but researchers say these steps make leaks much less likely:

Use an enterprise account

Universities that license ChatGPT, Claude, or Gemini typically sign contracts that bar the company from training on user data, whereas consumer accounts get no such protection. If “you’ve been using a personal account for your research, switch to your university’s enterprise account,” says Jennifer King, privacy and data policy fellow at Stanford University.

Even with an enterprise account, however, “no training” doesn’t always mean “no use.” California State University’s contract with OpenAI, shared by CalMatters, lets the company “process information derived from [customers] that has been deidentified, anonymized, and/or aggregated.” King suggests researchers check the fine print of their institution’s AI deals.

Turn off training on personal accounts

If you use a personal account, switch on the setting that stops your chats from being used for training.

Watch for exemptions

On ChatGPT, rating a response with a thumbs up or down “can make the entire associated conversation eligible for training,” even if you’ve opted out, Mireshghallah notes in a recent analysis of company policies. Anthropic keeps feedback data, too, including the related conversation, for 5 years, and may use it to train AI models. Other providers have similar policies.

Chats that AI systems flag as potentially unsafe, such as those touching on bioweapons or mental health, are another possible carve-out. For example, when Anthropic’s systems flag a chat, the company keeps the inputs and outputs for up to 2 years, and other kinds of safety-related data for up to 7 years

Practice data hygiene

Once a project is complete, King suggests downloading your chat record, “then maybe delete it.” A log of your AI use may be needed for replication or disputes, but it needn’t stay with your provider. Deleting isn’t watertight. Anthropic, for example, says deleted chats leave its servers within 30 days, but it can hold onto them longer for safety reasons. And deletion doesn’t remove data from models already trained on them.

Host your own model

For those savvy enough, the most private option is running an open-weight model on your own hardware. But that does mean “moving off of some of these platforms that host it for you, have nice tooling, and do a lot of things as part of the product,” Cooper says.

If it’s really sensitive, consider not using AI

The decision often boils down to weighing potential risks against the rewards. But even “not sending stuff [to AI] yourself will only get you so far,” Cooper says, pointing out that a peer reviewer or colleague could run a manuscript through AI even if you don’t.

As for Buckmaster and Mestre, both still use AI, but want much more control over where their work goes. “I still use AI extensively and believe it can be enormously useful,” Mestre says. “What has changed is how much control and visibility I want over the data, infrastructure, and actions of the agent.” Academics, Buckmaster says, “need to somehow push back and take some sort of ownership of our user data.”

Read More