Abstract:
DeepSeek has also joined the parameter competition. According to The Information, DeepSeek is currently training a new model with approximately 2 trillion parameters; Liang Wenfeng also stated at a recent investor meeting that the company’s next step is to continue pushing the model to 8 trillion parameters. A year ago, 8 trillion was still an unimaginable number.
But now, Kimi K3 has reached 2.8 trillion parameters; Byte is pre-training a new model that may reach up to 10 trillion parameters; Alibaba has also publicly stated that the next generation model will push towards 5 trillion to 10 trillion parameters. Anthropic does not disclose model parameters, but FT has previously quoted industry estimates as saying that its latest flagship Mythos 5 has approximately 8 trillion parameters.
The big model went around in a circle and started competing again to see who was bigger.
Only this time, the parameters that everyone rolls out are no longer the same parameters as in the previous round of Scaling.
When DeepSeek released V4-Pro in April this year, the officially disclosed scale was 1.6 trillion total parameters and 49 billion activation parameters.
The follow-up plan is RMB 8 trillion, which is equivalent to 5 times that of V4-Pro.
This is actually quite unusual when it comes to DeepSeek. After all, DeepSeek left the deepest impression on the entire AI industry last year, not in terms of parameters.
It once caused the AI industry to re-discuss that perhaps cutting-edge models do not need to burn GPU infinitely.
When V3 is released at the end of 2024, DeepSeek deliberately puts the training bill on the table: 671 billion total parameters, 37 billion parameters activated per Token, and complete training consumes 2.788 million H800 GPU hours. Calculated at US$2 per GPU hour, the official training cost is only US$5.576 million.
This number, and the efficiency behind it, became one of the most important labels for DeepSeek later.
DeepSeek does not have an abundant supply of advanced GPUs like OpenAI and Anthropic; the company has long relied on Liang Wenfeng’s Huanfang to quantify its self-transfusion and does not accept external financing. Its conditions determine its route - since money and computing power are limited, performance should be maximized from architecture, training and inference efficiency.
But now, the conditions have changed.
DeepSeek just completed its first round of external financing since its establishment in June this year, raising approximately US$7.4 billion. Currently, the second round of financing of approximately RMB 50 billion (approximately US$7.5 billion) has entered the finalization stage, corresponding to a valuation of approximately RMB 500 billion. If this financing is successfully completed, DeepSeek's cumulative external financing will be close to 15 billion US dollars and approximately 100 billion yuan within a few months.
At the same time, DeepSeek has hired CITIC Securities to prepare for the IPO of the Science and Technology Innovation Board, and the new capital will be clearly used for computing infrastructure, model development and talent investment; on September 21, Yan Wentao, a former Hillhouse Venture Partners partner, officially joined DeepSeek and served as CFO, ending the CFO vacancy in the three years since the company was founded.
In the past, DeepSeek used limited money and computing power to make the model as good as possible; now financing and listing are being pushed forward. With more money in hand, the computing power can also be increased.
After conditions are met, DeepSeek also begins to boldly roll out parameters.
This does not conflict with the pursuit of efficiency. When efficiency is high, the scale can be expanded with the same money.
In the past two months, the total parameter scale of cutting-edge models has been significantly increasing.
Pushing this competition back into the spotlight is The Dark Side of the Moon. In July this year, Kimi K3 achieved 2.8 trillion total parameters and 104 billion activation parameters. K3 is still the open weight model with the largest parameter scale that has been officially released.
It's not just big. In the official evaluation, K3's GPQA Diamond reached 93.5, and Terminal-Bench 2.1 reached 88.3, which is very close to GPT-5.6 Sol and Claude Fable 5 in the same period. The scale has reached trillions, and the performance has reached the first echelon.
In August, Byte was revealed to be planning to become bigger.
On August 6, "LatePost" first disclosed that Byte was discussing training a new model with more than 5 trillion parameters, led by Xiang Liang, head of the Seed Foundation. According to reports, Byte internally seems to believe that instead of continuing to catch up on the existing size, it is better to push the parameter scale several times higher than that of its peers at once.
One day later, on August 7, FT reported, citing people familiar with the matter, that the scale of the new model being pre-trained by Byte may reach up to 10 trillion parameters; when Reuters followed up, it also compared it with Anthropic's cutting-edge Mythos - although the latter has never disclosed the parameter scale, industry estimates have reached about 8 trillion. If Byte's new model reaches 10T, it will exceed the scale currently circulated by Mythos.
Now, extremely large parameter quantities of more than 5T have begun to appear repeatedly in discussions of cutting-edge models.
Today, Alibaba CEO Wu Yongming publicly announced that the next generation model plans to achieve 5 trillion to 10 trillion parameters; DeepSeek is training a 2T model while setting the follow-up target to 8T. It can be understood that the head laboratory has once again regarded a larger total parameter scale as an important route to impact the upper limit of capabilities.
Even American closed-source models that do not disclose parameters cannot escape outside scrutiny. Anthropic's Mythos is estimated to be 8T by outsiders; OpenAI has no longer disclosed the model scale since the GPT-4 era, but there is a rumor in the community that Astra has reached a 10T MoE.
The number of parameters has never lost its appeal, just like the combat power value on the game panel - people know that it cannot represent everything, but it will always be the most intuitive and oppressive number.
In fact, the large model industry has not been very fond of talking about parameters for some time.
In the past two years, distillation, small models, MoE, reinforcement learning and inference time calculations have taken turns. Rather than showing off that they have hundreds of billions or trillions of parameters, model manufacturers are more willing to talk about performance, cost and efficiency. Simply making the model bigger will easily make it look like you have money but no place to spend it.
Today, parameter quantities have been raised again and have once again become the "front" of cutting-edge models.
However, today's 5T, 8T, and 10T are no longer the same thing as the "more parameters, the larger the model" in the previous round of Scaling.
The most direct reason why parameter quantities can be put back on the table is that the mainstream architecture of ultra-large models has changed.
In the past, the mainstream of large models was Dense architecture, where parameter scale and calculation amount were basically tied together. Simply put, it means how many parameters there are in the model, and they basically have to be entered together in every calculation. The more parameters there are, the higher the upper limit of capabilities tends to be, but the cost of training and inference also increases.
Nowadays, more and more large models use MoE, Mixture of Experts, and mixed expert architecture.
It is more like a company with many experts. The model can have hundreds or thousands of experts, but only a small part of them are selected to work on each task. For example, Kimi K3 has 2.8T total parameters, but each Token only activates 104B, about 3.7%; DeepSeek V4-Pro has 1.6T total parameters, and each Token only activates 49B, about 3%.
So today, if a model has 5T, 8T or even 10T parameters, it does not mean that it must run through these 10T parameters every time it generates a Token.
How much the model can hold and how much needs to be calculated each time have become two different numbers.
Parameters can be understood as the space used by the model to carry knowledge, patterns and capabilities. The larger the total parameters, the more space the model theoretically has to learn more and more complex distributions; MoE allows it to only call some of them when needed.
As a result, the model can continue to increase the total parameter capacity without making each calculation heavier year-on-year.
Still taking K3 as an example, it has a total parameter of 2.8T, which is about three times the scale of K2.5, but among 896 experts, only 16 of each Token are activated. Dark Side of the Moon said that thanks to sparser MoE and a series of architectural optimizations, the overall scaling efficiency of K3 has been improved by about 2.5 times compared to K2.
The Agent era has just pushed this "capacity" back to the center of competition.
In the past, the ability of a Chatbot was often measured one by one. Mathematics looks at mathematics, code looks at code, and Q&A looks at knowledge. If the model reaches higher peaks on several key benchmarks, it is enough to prove that it is very strong.
But Agent strings these capabilities into the same task chain. A real long task may start with searching for information, then read web pages, look at pictures, write code, call the terminal, and then encounter errors, modify the plan, call other tools, and even assign subtasks to other agents.
When OpenAI released the Agents API in September this year, it directly listed long-term running, context management, tool usage and sub-agent coordination as the core infrastructure of the Agent, and emphasized that the Agent needs to run reliably for hours or even days.
In a Q&A, if a certain ability fails occasionally, only one question may be lost; in an Agent task involving dozens or hundreds of steps, any weak link may interrupt the entire link. The longer the task, the higher the requirements for capability coverage and stability. As a result, frontier competition has begun to have an obvious "barrel effect."
METR now even directly uses "task time span" to measure Agent capabilities, because the longer the task, the higher the requirement for the model to continuously complete a series of different operations.
In the Chatbot era, it is easier to see the peak of capabilities, while in the Agent era, capability coverage is increasingly important.
At this time, larger parameter capacity has new value. The longer the task, the less biased the model must be; the more things the Agent can do, the wider the capabilities that the base needs to cover.
MoE opens up room for continued expansion of capacity, and Agent further amplifies the value of capacity. Therefore, model manufacturers are trying their best to improve efficiency while pushing the parameters to 5T, 8T, and 10T again.
The saved computing power did not make Scaling disappear, but instead made new space for Scaling.
Comments