NVIDIA Introduces Reshard Technique to Cut Weight Transfer Time in Reinforcement Learning

At a glance

In reinforcement learning post-training, the latest model weights must be continuously moved from training GPUs to inference GPUs. This transfer is called refit, and as models have grown, the amount of data that must be moved at once has increased sharply. NVIDIA introduced Reshard, a new communication method that moves only the necessary data.

Key points

1. 1

   The existing method combined all shards once and then broadcast them whole, causing many unnecessary transfers.

2. 2

   Reshard compares the layouts of the sending and receiving sides and moves only the parts that are needed.

3. 3

   On NVIDIA DGX H100 and GB200 systems, it was measured to be 2.7 to 5.7 times faster than the existing method.

4. 4

   A hybrid approach is also used: Reshard is applied to a subset of weights that make up most of the total data, while the existing method is kept for the rest.

Why it matters

This technology reduces the time and cost of fine-tuning large AI models with reinforcement learning, so it could affect the speed of AI development.