Abstract:
Affected by the continued tight supply of DRAM, the AI industry is shifting from "computing power limited" to "memory limited". According to reports, Google has begun to dismantle decommissioned servers, recycle the DDR4 memory modules that can still be used, and reuse them in some newly built AI servers, gradually establishing an internal memory recycling supply chain.
Nikhil Cherian, Google's senior director of supply chain infrastructure, recently revealed at a summit forum that high-performance memory currently accounts for about 75% of the bill of materials cost of an AI server. In order to break through the memory bottleneck, Google is promoting both software and hardware-level solutions, including dismantling decommissioned servers and recycling usable components.
Google has even specially designed hardware adapters to enable previous generations of memory solutions such as DDR4 to be connected to new generation AI servers. Cherian also admitted that Google will specifically repatriate some retired servers just to remove the DDR4 memory modules.
At the same time, Google is also continuing to optimize the underlying function library, model architecture, and KV cache compression technology to reduce the memory capacity required per unit of computing power.
Google launched two TPU ASIC versions this year, of which TPU8t is mainly used for AI model training, while TPU8i is optimized for inference tasks. To eliminate performance bottlenecks, TPU8i adopts a highly customized multi-layer memory design.
Each TPU 8i chip is equipped with 288GB HBM3e memory, which can provide ultra-high memory bandwidth of 8.6TB per second. The chip also integrates 384MB of on-chip SRAM for storing an active key-value cache to avoid frequent access to external system memory.
In the server host part, the TPU 8i server completely cancels the traditional x86 main processor and uses Google's own customized Axion processor based on the Arm architecture. The Axion processor relies on the high-speed DDR5 memory architecture and is mainly responsible for host-level tasks such as data preprocessing.
However, based on Cherian’s latest statement, Google seems to have used older DDR4 memory in some new AI servers to replace the DDR5 modules that were originally supposed to be equipped. The specific server range and number of DDR4 modules involved have not yet been disclosed.
Comments