Faced with the emergence of more and more competitors in the AI server chip track, NVIDIA has come up with a response plan: cooperate with competing products. Industry media The Information exclusively learned that Nvidia and AI chip startup d-Matrix will integrate the hardware of both parties to create a new computing system for large model inference.
Nvidia CEO Jensen Huang
Just one month before this cooperation was revealed, another AI server chip manufacturer, SambaNova, announced that it had opened up software and hardware with NVIDIA so that its own chips can run AI models in conjunction with NVIDIA GPUs.
NVIDIA has released a signal that more similar cooperation projects will be launched in the future. Dionne Harris, senior director of high-performance computing at Nvidia, said: "Other cooperation is not disclosed in advance."
Chips from manufacturers such as d-Matrix and SambaNova can directly connect to Nvidia GPUs through universal Ethernet cables and are compatible without cooperation. However, in this in-depth cooperation, Nvidia engineers will jointly debug the GPU control software with competing chip teams, greatly improving cross-chip collaboration efficiency; formal cooperation binding will also make it easier to impress customers to purchase this hybrid computing machine.
This style of play fully reflects CEO Huang Jensen’s strategic adjustment: Nvidia’s leading share in the AI chip market continues to be impacted by many parties. Instead of fighting with potential opponents, it is better to actively join forces - once the products of emerging chip manufacturers are successfully commercialized, Nvidia can also share the revenue.
Thomas Sommers, chief technology officer of Positron, an AI chip start-up that has not yet cooperated with NVIDIA but is open to it, commented: "NVIDIA's strategy and industry perception have obviously changed. Now NVIDIA chooses to proactively show its goodwill and build a heterogeneous computing power ecosystem instead of doing its best to suppress competitors."
Server racks integrating NVIDIA GPUs and d-Matrix Corsair AI acceleration chips have been deployed in AI inference cloud service provider Gimlet Labs
The Multiple Values of a Collaborative Strategy
On the one hand, this strategy can resolve past market accusations: some customers once complained that Nvidia coerced manufacturers into purchasing only its own hardware. The Information previously reported that the U.S. Department of Justice launched an investigation into the matter two years ago, but no formal charges have been filed.
Harris said that Nvidia’s long-term positioning is a comprehensive AI infrastructure service provider, “We are by no means just a chip company.”
NVIDIA's process of broadening its ecological layout has accelerated since last year: NVIDIA announced the opening of the NVLink high-speed interconnect interface, allowing third-party chips to be networked with it to improve the efficiency of server cluster collaboration. Even if chip sales do not increase, Nvidia can use this to sell more network interconnection hardware.
Harris said bluntly: “It is better to have a product to sell than to have nothing at all.”
In December last year, Nvidia spent $20 billion to obtain technology licenses from inference chip manufacturer Groq and absorb its core R&D team. The deal is similar to a disguised acquisition, but Groq remains an independent operation; Nvidia simultaneously develops dedicated server racks that integrate Nvidia GPUs and Groq chips. At present, the market demand for this complete product is still unclear.
A close second was the partnership with SambaNova. Rodrigo Liang, CEO of SambaNova, said that Nvidia views the positioning of this start-up as far more than an ordinary competitor, but more as a partner.
Calculation data from The Information shows that despite the entry of companies such as Google and Amazon into the competition, Nvidia’s share of the AI inference chip market has increased instead of falling in the past two years. Huang Renxun has also emphasized many times that the overall efficiency of NVIDIA GPUs in completing various reasoning tasks is better than that of competing products; however, NVIDIA does not passively wait for the natural selection of the market. Instead, it relies on its strong balance sheet to provide financial support to small and medium-sized cloud manufacturers, lowering the threshold for customers to purchase high-priced NVIDIA AI chips, including the low-cost computing power leasing project.
At the same time, Microsoft, Meta, OpenAI, and the latest entrant Anthropic are all developing or deploying inference chips in-house, which may dilute Nvidia's market share in the long run.
An OpenAI spokesperson said that the company has not yet determined whether it will establish a chip co-operation solution with Nvidia. Nvidia has invested heavily in OpenAI, and the two parties are also negotiating to provide funding for its Ohio hyperscale data center project.
Sid Shays, CEO of d-Matrix, said: "In the future, the industry will move towards a multi-chip collaboration pattern. NVIDIA is fully able to accept the division of labor between different chips to handle different aspects of the same AI task. The era of a single GPU monopolizing computing power is gone forever."
In the past, emerging chip manufacturers generally regarded themselves as "NVIDIA disruptors," but NVIDIA has established insurmountable technical and ecological barriers. Start-ups now prefer to cooperate with NVIDIA GPUs rather than completely replace them. NVIDIA has also proactively extended an olive branch to such emerging companies for cooperation.
According to people familiar with the matter, this cooperation was prompted by Nvidia’s initiative to contact d-Matrix, which is also headquartered in Santa Clara, California.
For a long time, AI companies such as OpenAI have mixed different models of Nvidia GPUs to handle the same task - manufacturers have long discovered that low-computing power GPUs are more suitable for specific workloads. butCross-vendor heterogeneous chip collaborative reasoning, is still a new architectural idea that has only emerged in the past two years.
Nvidia is not the only one laying out heterogeneous inference. Amazon, which develops its own Trainium chip (for use by Anthropic and is about to launch OpenAI), announced in March this year that it will jointly launch an integrated AI inference server with AI chip manufacturer Cerebras.
Dismantling of heterogeneous reasoning technology
Split the large model inference process into different architecture chips for execution, which is called in the industryDisaggregated Inference.
Take NVIDIA's solution with SambaNova and Groq as an example: NVIDIA GPU is responsible forPrefill——The link with the highest computational power consumption in reasoning; chips from start-up manufacturers will take over the follow-upDecodeGenerate tokens.
The d-Matrix solution adopts a two-way division of labor: the two types of chips share part of the pre-filling and decoding work at the same time, and the d-Matrix chip focuses onSpeculative Decoding, greatly improving the response speed of large models.
After the customer issues a task instruction, the d-Matrix chip runs a small draft model to predict the output word units of the large model; the main large model equipped with NVIDIA GPU then verifies and accepts the prediction results to reduce the overall reasoning delay.
"Embracing a diverse chip ecosystem"
Sid Shays, CEO of d-Matrix, once again emphasized: "In the future, the industry will move towards a pattern of multi-chip collaboration. NVIDIA is fully able to accept the division of labor between different chips to handle different aspects of the same AI task. The era of a single GPU monopolizing computing power is gone forever."
d-Matrix was established in 2019 and completed a US$275 million round of financing in November last year, with a post-investment valuation of US$2 billion; three people familiar with the matter revealed that the company has launched a new round of financing negotiations.
Xie Si introduced that the core differentiation advantage of d-Matrix lies in integrating the computing unit and memory into a single chip, and does not use the same high-bandwidth memory (HBM) of Nvidia that is currently in short supply. TSMC has started mass production of d-Matrix chips this summer, and the company plans to produce thousands of chips per month by the end of the year.
He revealed that the company's current annual revenue is only a few million dollars; it aims to have its chips' total power consumption in data centers reach 30-40 megawatts next year, which can support AI code generation, voice and video reasoning services.
Parasail, an emerging AI cloud vendor in San Mateo, California, will become the first customer to purchase a joint server machine from NVIDIA and d-Matrix, and plans to open this hybrid computing power service to tenants in the second half of this year. Parasail CEO Mike Henry said that this set of combined servers can help enterprises get rid of their high dependence on single procurement of Nvidia hardware and is extremely attractive.
