The global AI computing power competition has reached a major technological turning point! Recently, Google announced the new AI memory compression technology "TurboQuant", which has attracted great attention in the industry. This technology claims to be able to reduce the space requirements of the most resource-intensive "key value cache" (KV Cache) in the generative AI inference stage to 1/6 of the original without sacrificing model accuracy, and increase the calculation speed by 8 times.
This breakthrough technology has also triggered concerns in the entire market that memory demand will fall off a cliff. Storage-related U.S. stocks such as Micron, Sandisk, and Western Digital have plummeted.
What exactly is TurboQuant?
During the LLM (Large Language Model) inference process, in order to process long texts, the system must store past conversation information in the KV Cache, which is like an AI "portable notebook". As the length of the conversation increases, the information that needs to be stored in this notebook will quickly overwhelm the high-bandwidth memory (HBM) of the AI GPU, becoming the biggest bottleneck in AI operation.
The core advantage of Google's TurboQuant technology is to solve the "memory noise" (Overhead) produced by traditional memory compression technology. The technology consists of two key parts:
PolarQuant (polar coordinate quantification): Traditional vectors are marked with XYZ coordinates, and the operation is cumbersome. Google instead uses "polar coordinates" logic to simplify complex orientations into "radius" and "angle". This is like simplifying the information that was originally meant to be labeled "walk 3 kilometers east and 4 kilometers north" to "walk 5 kilometers at an angle of 37 degrees." This transformation of geometric structures greatly reduces the data processing load.
QJL (Quantized Johnson-Lindenstrauss): This is an extremely streamlined 1-bit mathematical correction mechanism. Only an additional 1 bit is used to accurately correct the residual error in the compression process, so that even if the model is compressed to only 3 bits, it can still achieve "zero precision loss" in many benchmark tests such as LongBench.
△On the Llama-3.1-8B-Instruct model, TurboQuant demonstrated strong KV cache compression performance in the LongBench benchmark test, outperforming various compression methods (the bit width is indicated in parentheses).
Google chose to fully open source this set of technologies that have become core competitiveness. It not only optimizes the retrieval efficiency of large models such as Gemini, but also reduces the dependence on memory for other large models and paves the way for accelerating the development of end-side AI.
According to actual measurements, on the NVIDIA H100 accelerator, TurboQuant's performance is improved by up to 8 times compared to the uncompressed solution, and the model can be mounted directly without retraining. It can be called a "magic weapon" for reducing costs and increasing efficiency in AI deployment.
△On the NVIDIA H100 accelerator, TurboQuant shows significant performance improvements in computing attention logic values in the key-value cache, outperforming the highly optimized JAX baseline at various bitwidth levels.
△TurboQuant demonstrates strong retrieval performance, achieving the best 1@k recall on the GloVe dataset (d=200) relative to various state-of-the-art quantization baselines.
Cloudflare CEO Matthew Prince and others have called TurboQuant Google's "DeepSeek moment" and believe that it is expected to, like DeepSeek, significantly reduce the operating costs of AI through extremely high efficiency gains while remaining competitive in terms of results.
Will memory requirements decrease, or create greater requirements?
Regarding TurboQuant technology, which has triggered concerns about a sharp decline in memory demand in the entire market, industry experts and research institutions have also given completely different views:
Wells Fargo analyst Andrew Rocha noted:"When the context window (context window) is getting larger and larger, the explosive growth of KV Cache was originally a guarantee to push up the memory demand. But TurboQuant is directly attacking this cost curve. Once widely adopted, the specification requirements for memory capacity in data centers will be marked with big questions."
However, well-known investment bank Morgan Stanley and research organization Lynx Equity Strategies gave completely different views.
Morgan Stanley believes that the market may have ignored the economic law that "efficiency improvements drive aggregate growth."When the memory cost required for AI computing is reduced to 1/6 of the original, this will cause a massive explosion in demand for AI applications that were previously unable to go online because memory was too expensive (such as long text translation, complex code generation), and will instead fill or even exceed the compressed memory gap.
This is Jevon's paradox, which is that when technological progress improves the efficiency of using resources (reducing the quantity required for any one use), but the cost reduction leads to an increase in demand, the speed of resource consumption increases instead of decreasing.
Morgan Stanley analyst Joseph Moore and his team noted in an investor note released on Thursday: "There are reports that Google's TurboQuant will reduce memory usage to 1/6 of the original, but this ignores that they are only referring to KV Cache, not overall memory usage.
“It is worth noting that both Google’s Gemini 3 and 2.5 Pro models have context windows of 1 million Tokens, but Google has revealed that they have tested context windows of up to 10 million Tokens using Gemini 1.5 Pro and achieved very good results, but due to the high cost of inference, they ultimately did not release the model,” Moore said. "As a result, we expect that as this type of innovation and other technologies emerge, costs will decrease and this technology will be used to serve smarter, compute-intensive products."
Morgan Stanley further pointed out that TurboQuant mainly optimizes the cache in the "inference phase", not the model weights in the "training phase". Therefore, the impact on the HBM (high bandwidth memory) procurement logic that supports AI core training is relatively limited.
In contrast, TurboQuant is more meaningful for the deployment of artificial intelligence on terminal devices such as mobile phones and laptops. Since mobile devices have limited memory, this type of efficient compression technology allows more powerful AI models to run on mobile phones, which in turn will stimulate a comprehensive upgrade of memory specifications in various terminal devices.
According to Lynx Equity Strategies,While AI providers will need innovation to address bottlenecks in inference that arise as token context length increases, this will not reduce demand for memory and flash over the next three to five years due to supply constraints.