On Thursday, November 5, Eastern Time, NVIDIA (ASX: NVDA) published a company blog post detailing its full-stack co-design strategy for AI memory, emphasizing that as large model parameter scales, context lengths, and agent applications continue to expand, memory pressure on AI infrastructure is intensifying, and simply increasing GPU memory capacity is no longer enough to solve the problem.
NVIDIA (NVDA) proposed that by co-designing GPUs, CPUs, networks, storage, and software, and combining techniques such as model weight quantization, KV Cache compression and tiered storage, and retrieval data optimization, the memory utilization efficiency of the entire AI infrastructure can be improved. The company disclosed that from the Blackwell Ultra to the Vera Rubin platform, the HBM capacity of a single GPU increases by only about 3%, yet under specific agent workloads, the throughput achievable per dollar can rise by as much as 30 times.
The core of this strategy is to treat the memory and storage of an AI factory as a single unified data hierarchy, rather than obsessively trying to cram more data into the fastest and most expensive GPU memory, so that more AI tasks can be processed at a lower unit cost.
It is worth noting that on the same day NVIDIA (NVDA) published this technical article, the company's stock suffered a severe sell-off. During Thursday's session, the share price fell more than 9% at one point, erasing its year-to-date gains and turning negative, with market value evaporating nearly $1.4 trillion from the record high set in October.
Behind this sell-off, the market is re-examining the return cycle of AI infrastructure investment, the pace of data center construction, and the valuations of companies in the AI industry chain, highlighting a market environment in which the AI industry is shifting from chasing computing power expansion to focusing on investment efficiency. The full-stack co-design that NVIDIA (NVDA) emphasized this time is precisely an attempt to respond to the resource and cost constraints facing continuous AI infrastructure expansion from the perspective of system efficiency and cost per token.
Large Models and Agents Increase Memory Pressure, Simply Adding HBM Capacity Is Not Enough
NVIDIA (NVDA) pointed out that the ever-growing parameter scale of large models, increasingly long context windows, and the need for agents to continuously retain historical information across multiple rounds of inference and tool calls are putting greater pressure on GPU high-bandwidth memory (HBM).
Among these, KV Cache (key-value cache) is an important source of memory consumption during inference. It stores context information that has already been computed when the model processes a request, so that the model does not need to repeatedly process the same information when generating subsequent content. For agents, a single task may involve multiple rounds of model inference, tool calls, and result returns, and historical context will continuously accumulate, causing the KV Cache to expand accordingly. NVIDIA (NVDA) said that the scale of processing context involved in some agent task trajectories has already reached 3 million tokens.
In addition, retrieval-augmented generation (RAG) technology requires converting enterprise documents into vector representations and retrieving relevant content during model inference, which also increases the amount of data that needs to be processed and retained.
NVIDIA (NVDA) emphasized that during inference, both model weights and the KV Cache need to occupy HBM, and HBM is precisely the fastest, most expensive, and most supply-constrained memory tier in an AI factory.
Simply increasing HBM capacity cannot solve all problems. NVIDIA (NVDA) compared HBM to a warehouse and memory bandwidth to the warehouse's loading dock: a larger warehouse can hold more goods, but if loading and unloading speed cannot keep up, adding storage space will not improve actual throughput.
In AI inference, more HBM can indeed accommodate model workloads and KV Cache for more users, but all users still need to share GPU compute resources and memory bandwidth. Once compute capability or bandwidth becomes the bottleneck, the additional memory capacity may not be fully utilized, and may instead increase cost and power consumption.
Therefore, NVIDIA (NVDA) believes that what truly matters is not simply increasing memory capacity, but placing different types of data into the most appropriate storage tier and bringing it back to the GPU in a timely manner when needed, so as to improve the utilization efficiency of the entire system.
NVFP4 Compresses Model Weights, NVLink Distributes Weights Across 72 GPUs
To address the problem of model weights occupying large amounts of HBM, NVIDIA (NVDA) proposed compressing model weights through the NVFP4 low-precision format and using NVLink to achieve distributed storage across GPUs.
Specifically, NVFP4 can compress model weights to 4-bit precision, further reducing storage space compared with FP8. NVIDIA (NVDA) said that more and more model developers have begun to release model weights in NVFP4 format, including some models from institutions and companies such as Black Forest Labs, Thinking Machines Lab, Reflection, and Meta.
At the hardware level, NVIDIA (NVDA) uses NVLink to build the NVL72 rack-scale GPU interconnect system, allowing model weights to be distributed across up to 72 GPUs, so that a single GPU only needs to store part of the model, thereby freeing more HBM space to accommodate KV Cache and serve more concurrent users.
This means that NVIDIA (NVDA) is trying to solve the problem of model weights occupying memory through low-precision quantization and rack-scale interconnect together, rather than relying solely on continuously increasing HBM capacity on a single GPU.
KV Cache Compression and Tiered Offloading: Vera CPU and Dynamo Jointly Manage Memory
Compared with model weights, the challenge brought by KV Cache is more dynamic. As agents continue to reason, call tools, and wait for external results, part of the context needs to be retained, but not all information needs to always occupy the fastest GPU memory.
In response, NVIDIA (NVDA) proposed optimizing the KV Cache from three aspects: compression, temporary staging, and tiered offloading.
First, NVFP4 can also be used to compress the KV Cache, reducing its memory footprint compared with FP8 and allowing more context to remain in the GPU's HBM.
Second, the Vera CPU is responsible for accelerating tool calls, shortening the time during which the KV Cache sits idle while the agent waits for external tool execution results. At the same time, the Vera CPU is equipped with large-capacity LPDDR memory and connects to the Rubin GPU through NVLink-C2C, providing storage space closer to the GPU for temporarily inactive context, which can then be quickly brought back when needed.
Third, NVIDIA (NVDA) Dynamo inference software can offload temporarily inactive KV Cache from GPU memory to the CMX context memory storage platform. CMX is a tier designed for large-capacity, low-cost data storage, with a per-bit storage cost far lower than HBM, and it can bring data back before the context is called again.
Unlike local SSDs, CMX can be shared across servers; unlike traditional enterprise storage, it does not need to bear the additional overhead of long-term data protection for these short-lived contexts.
NVIDIA (NVDA) also uses Dynamo's KV Cache Reuse feature to manage KV Cache in GPU memory, CPU memory, and storage as a unified system-level resource, enabling already-computed context to be reused throughout the system and reducing repeated computation.
NeMo Retriever and cuVS Optimize RAG, Enterprise Knowledge Bases No Longer Occupy HBM
In addition to model weights and KV Cache, NVIDIA (NVDA) also proposed optimization solutions for the memory pressure brought by retrieval-augmented generation (RAG).
RAG needs to convert enterprise documents into vector embeddings and build a searchable vector index so that the model can extract relevant information from a massive knowledge base based on user questions. If all of these vector data are placed in GPU HBM, they will further squeeze the valuable memory needed for model inference.
NVIDIA (NVDA) proposed optimizing the storage of vector data and indexes through NeMo Retriever and cuVS respectively.
Among them, the NeMo Retriever model can generate smaller vector embeddings while improving retrieval accuracy and reducing the amount of data that needs to be stored and transmitted; cuVS supports building and running large vector indexes in CPU memory and storage tiers, so that the entire enterprise knowledge base does not need to occupy GPU HBM.
Only the relevant text fragments that are actually retrieved and need to be processed by the model will enter the model context.
Through this approach, NVIDIA (NVDA) hopes to reduce the HBM footprint of model weights, idle context, and RAG data, leaving more GPU memory for actual inference tasks and concurrent users.
From Expanding Memory to Improving Output per Unit of Resource, Vera Rubin Efficiency Up to 30x Higher
The performance data disclosed by NVIDIA (NVDA) is the most noteworthy part of this full-stack co-design article.
The company cited the DeepSeek V4 Pro workload in the SemiAnalysis AgentX benchmark to compare the throughput and cost efficiency of different generations of AI computing platforms.
Specifically, at a service speed of 100 tokens per second per user, NVIDIA (NVDA) GB300 NVL72 achieves up to 10 times higher throughput per dollar compared with the H200 platform, while the HBM capacity of a single GPU roughly doubles.
At a service speed of 125 tokens per second per user, Vera Rubin NVL72 achieves up to 30 times higher throughput per dollar compared with GB300 NVL72, while the HBM capacity of a single GPU increases by only about 3%.
This means that in the specific agent inference scenarios emphasized by NVIDIA (NVDA), the improvement in performance and cost efficiency far exceeds the gains that can be brought by simply increasing memory capacity. By compressing model weights and KV Cache, offloading idle context to lower-cost storage tiers, and optimizing data scheduling among GPUs, CPUs, and networks, the system can obtain more effective computing output from memory resources of similar scale.
NVIDIA (NVDA) emphasized that for AI factory operators, the truly important metric is not simply how many GPUs or how much HBM they have, but how many useful tokens can be delivered within a given power budget while meeting user response speed requirements. Generating more tokens per watt means the same power resources can complete more work and create more revenue, while also lowering the cost per token.
Comments