Abstract:
Groq, an AI inference chip company that has been acquired by NVIDIA, is planning to introduce 3D stacked DRAM technology into its next-generation LPU (Language Processing Unit) products to replace the currently widely used SRAM solution. This adjustment means that the company will focus more on increasing chip storage capacity in the future to adapt to the expanding demand for large-scale artificial intelligence models.
Earlier this year, NVIDIA announced the Groq LPX rack product and included it in the Vera Rubin ecosystem to enhance AI inference performance. The rack entered full mass production in August this year. According to early test results, when working in conjunction with the NVL72 server, the system can achieve an inference speed of 3,400 Tokens per second, setting a new performance record.
The current generation of Groq LPU is designed with 500MB SRAM and has a memory bandwidth of up to 150TB/s. In the LPX rack, the system is configured with 256 of these chips, providing a total of 128GB of SRAM capacity and 640TB/s of expanded bandwidth capabilities. A single Groq LPU can provide about 1.20 PFLOPs of FP8 computing power.
But for the next generation of products, Groq is preparing to take a completely different route. The company plans to abandon its reliance solely on SRAM and instead adopt DRAM and combine it with 3D Stacked (3D stacked) packaging technology to create a new architecture. Compared with SRAM, the biggest advantage of DRAM is that it can provide much higher storage capacity than SRAM, but at the expense of relatively low bandwidth and access speed.
A semiconductor industry source said that Groq has plans to introduce 3D packaging technology in the next generation of chips, and the stacking object will change from existing SRAM to DRAM. The move is widely interpreted in the industry as Groq's desire to significantly increase chip storage capacity. Although SRAM is extremely fast, its capacity expansion is limited; in comparison, although DRAM is slower, it is more suitable for building large-scale storage systems.
Using 3D stacking technology, Groq can stack multiple layers of DRAM in a single vertical structure chip and utilize TSV (Through Silicon Via) technology to achieve inter-layer connections. This design concept is very similar to how current high-bandwidth memory HBM is implemented. Compared with traditional DRAM packaging, the 3D stacking solution can not only significantly increase the storage density per unit area, but also improve energy efficiency and reduce space requirements.
Judging from the information disclosed so far, the focus of Groq's product upgrade has obviously shifted to storage capacity, while still hoping to retain as much as possible some of the high-bandwidth advantages brought by its SRAM architecture. As the scale of generative AI models continues to increase, the need for large-capacity, high-speed memory continues to grow, so the new generation of Groq chips is expected to be more widely used.
The report also mentioned that Groq may not abandon SRAM completely in the future, but will develop DRAM and SRAM product lines in parallel to provide differentiated solutions according to different application scenarios. For NVIDIA, which is building AI factories and AI platforms, this type of higher-capacity inference chips are also expected to play a more important role in the future.
Comments