When something becomes wildly popular, it becomes valuable. Then, it's only a matter of time before someone tries to pirate it, nullify the barriers around it, or outright clone it.
Google's Gemini, one of the most advanced AI-powered chatbots out there, has now joined that inevitable club.
In a report from the Google Threat Intelligence Group, the company revealed that commercially driven actors bombarded Gemini with over 100,000 prompts in a sophisticated effort to distill and replicate its core capabilities.
These aren't that typical casual users testing limits. The attacks fall under what's called "distillation attacks" or model extraction, where adversaries systematically query the AI to map out its reasoning patterns, chain-of-thought processes, and decision-making logic. By collecting vast numbers of input-output pairs, they aim to train a smaller, far cheaper copycat model that behaves almost identically to the original, all without the billions in compute and data investment required to build one from the ground up.
Google described one particularly large campaign that spanned thousands of varied queries, often in non-English languages, to broaden the stolen model's reach and versatility.
In some cases, prompts were engineered to force consistent chain-of-thought outputs, even specifying that "the language used in the thinking content must be strictly consistent with the main language of the user input."
Google views this as straight-up intellectual property theft and a clear breach of its terms of service.
The perpetrators appear to be private companies and researchers chasing a shortcut in the fierce race for AI dominance, rather than state-sponsored groups in this instance, though the company has separately noted state-backed actors misusing Gemini for other malicious ends, like reconnaissance or malware development.
Google detected the massive probing effort, disrupted associated accounts, and quietly strengthened Gemini's defenses, though it hasn't shared specifics on those countermeasures.
This isn't a one-off problem.
The industry has seen echoes before, including OpenAI's accusations against rivals using similar techniques. As frontier models like Gemini remain publicly accessible through apps, APIs, and web interfaces, complete with rate limits that determined attackers can still work around, the vulnerability persists.
John Hultquist, chief analyst at Google's Threat Intelligence Group, put it bluntly: "We're going to be the canary in the coal mine for far more incidents."
He warned that the same risks will soon hit smaller organizations deploying custom large language models trained on sensitive proprietary data.
"Let’s say your LLM has been trained on 100 years of secret thinking of the way you trade. Theoretically, you could distill some of that," he explained, highlighting how distillation could leak hard-won competitive edges.
With AI adoption exploding and the economic stakes skyrocketing, unauthorized model cloning represents a new frontier in corporate espionage and IP battles. Google is mitigating where it can, but the fundamental tension remains: the more powerful and open a model is to users, the more it's exposed to those determined to reverse-engineer it.
As the arms race in AI continues, expect more stories like this one—because success breeds imitation, and imitation in this space often starts with a flood of cleverly crafted prompts.
