NVIDIA Explains Five Co-Design Methods to Get More Value From Memory in AI Factories

At a glance

In data centers that run AI services, how usefully companies use high-bandwidth memory (HBM, fast memory attached next to the GPU) has become important, because it is fast but expensive and scarce. NVIDIA says it is solving this problem by designing chips, software, CPUs, and storage together.

Key points

1. 1

   NVFP4 is a format that reduces numbers to 4 bits, which greatly shrinks the memory a model takes up.

2. 2

   When expert models are spread across several GPUs, each GPU has more memory left over for cache.

3. 3

   The Dynamo software splits conversation memory (the KV cache) across GPUs, CPUs, and storage, so multiple servers can share it.

4. 4

   The large memory of the Vera CPU temporarily holds conversation memory for tasks waiting on tool calls.

Why it matters

In an era when AI works in multiple steps, how much work can be done with the same memory determines the cost and speed of AI services.