China’s DeepSeek The Difficulty of Preventing AI Model Distillation

Leading White House advisers have raised concerns that China’s DeepSeek may have leveraged a technique called “distillation” to benefit from the advancements of US AI competitors.

Distillation allows a newer AI model to learn from an older, more powerful system, potentially bypassing the significant costs and resources required for training.

Industry experts in Silicon Valley argue that preventing this technique may be nearly impossible, especially with the rise of open-source models such as Mistral and Llama. These models are publicly available, making it difficult to regulate how they are utilized.

DeepSeek’s Disruptive AI Model

DeepSeek recently gained attention by launching an AI model that rivals US firms like OpenAI at a significantly lower cost.

Unlike its competitors, DeepSeek released its code for free, raising concerns that it might have drawn knowledge from US models to achieve its capabilities.

The process of distillation involves an established AI model evaluating the outputs of a newer model, effectively transferring its learnings.

While this technique has been widely used in AI research, it violates the terms of service of companies like OpenAI.

A spokesperson for OpenAI confirmed that the company is reviewing whether DeepSeek improperly used its models through distillation.

The AI Industry’s Competitive Nature

Naveen Rao, VP of AI at Databricks, noted that learning from competitors is common in the AI industry, comparing it to automakers examining rival engines.

He acknowledged that while firms aim to adhere to ethical guidelines, competition drives innovation and the adoption of advanced techniques.

Meanwhile, Howard Lutnick, the nominee for US Secretary of Commerce, expressed strong opposition to DeepSeek’s actions, pledging to enforce strict measures to maintain US leadership in AI.

Similarly, David Sacks, the White House’s AI and crypto policy lead, highlighted the risks of unauthorized distillation in a recent interview.

The Challenges in Blocking AI Distillation

Experts suggest that preventing distillation will be challenging, as DeepSeek demonstrated that even a small number of data samples from a larger model can significantly enhance a smaller model’s capabilities.

China’s DeepSeek With platforms like Llama and Mistral freely available, detecting violations of terms of service becomes increasingly difficult.

Meta’s Llama model requires users to disclose distillation practices, and DeepSeek has acknowledged using Llama for some models.

However, it remains unclear whether the company adhered to Meta’s licensing requirements throughout the process.

Potential Solutions and Ongoing Challenges

A source from a major AI lab suggested that the only way to curb distillation would be implementing strict “know-your-customer” requirements, similar to financial regulations. However, such measures are not yet in place, and future policies remain uncertain.

Some US AI firms, like Groq, have attempted to block Chinese IP addresses from accessing their cloud-based models.

However, CEO Jonathan Ross acknowledged that this approach is insufficient, as users can bypass such restrictions.

He emphasized that the challenge of preventing AI distillation is an ongoing “cat and mouse game.”

As AI competition between the US and China intensifies, the debate over distillation underscores the broader struggle to protect intellectual property while fostering innovation in a rapidly evolving field.