DeepSeek's new chatbot has made waves in the AI industry, positioning itself as a formidable competitor. The company introduced its AI with the intriguing tagline: "Hi, I was created so you can ask anything and get an answer that might even surprise you." This bold statement has resonated with users, and today, DeepSeek's advancements have contributed to one of the largest stock price drops for NVIDIA, highlighting the impact of its technology.
Image: ensigame.com
What sets DeepSeek's model apart is its innovative architecture and training methods. Here are the key technologies that power its AI:
Multi-token Prediction (MTP): Unlike traditional models that predict one word at a time, DeepSeek's MTP approach predicts multiple words simultaneously by analyzing different parts of a sentence. This method enhances both the accuracy and efficiency of the model.
Mixture of Experts (MoE): This architecture employs various neural networks to process input data. It accelerates AI training and improves performance. In DeepSeek V3, 256 neural networks are utilized, with eight being activated for each token processing task.
Multi-head Latent Attention (MLA): This mechanism focuses on the most significant parts of a sentence. MLA extracts key details from text fragments repeatedly, reducing the likelihood of missing important information. This ensures the AI captures crucial nuances in the input data.
Image: ensigame.com
DeepSeek, a prominent Chinese startup, claims to have developed a competitive AI model with minimal costs, stating they spent only $6 million on training the powerful neural network DeepSeek V3 and used just 2048 graphics processors. However, analysts from SemiAnalysis have revealed that DeepSeek operates a vast computational infrastructure comprising approximately 50,000 Nvidia Hopper GPUs, including 10,000 H800 units, 10,000 more advanced H100s, and additional H20 GPUs. These resources are distributed across several data centers and are utilized for AI training, research, and financial modeling.
The company's total investment in servers amounts to around $1.6 billion, with operational expenses estimated at $944 million. DeepSeek is a subsidiary of the Chinese hedge fund High-Flyer, which spun off the startup as a separate division focused on AI technologies in 2023. Unlike most startups that rent computing power from cloud providers, DeepSeek owns its own data centers, giving it full control over AI model optimization and enabling faster implementation of innovations. The company remains self-funded, which positively impacts its flexibility and decision-making speed.
Image: ensigame.com
Moreover, some researchers at DeepSeek earn over $1.3 million annually, attracting top talent from leading Chinese universities (the company does not hire foreign specialists). Even considering this, DeepSeek's recent claim of training its latest model for just $6 million seems unrealistic. This figure refers only to the cost of GPU usage during pre-training and does not account for research expenses, model refinement, data processing, or overall infrastructure costs.
Since its inception, DeepSeek has invested over $500 million in AI development. However, unlike larger companies burdened by bureaucracy, DeepSeek's compact structure allows it to actively and effectively implement AI innovations.
Image: ensigame.com
The example of DeepSeek demonstrates that a well-funded independent AI company can compete with industry leaders. Nevertheless, experts emphasize that the company's success is largely due to billions in investments, technical breakthroughs, and a strong team, while claims about a "revolutionary budget" for developing AI models are somewhat exaggerated. Still, competitors' costs remain significantly higher. For instance, compare the cost of model training: DeepSeek spent $5 million on R1, while ChatGPT4o cost $100 million.
