NVIDIA Recovery Automatically Restarts Stalled Distributed AI Training Jobs

At a glance

Large-scale AI training often stops because of hardware or software problems. NVIDIA Recovery is an open-source service that watches training jobs running on a cluster around the clock. When a stall happens, it finds the cause and handles recovery automatically. It is meant to reduce the burden of people checking logs day and night.

Key points

1. 1

   The service detects training stalls using application logs, GPU health metrics, and job scheduler signals.

2. 2

   After detection, it finds and isolates the faulty equipment, then restarts or stops the job from the saved checkpoint (a mid-training save point).

3. 3

   It can connect to Slurm (a job scheduling program). On first installation, it starts in monitor mode, which only shows recovery recommendations without taking any action.

4. 4

   From July 9 to October 9, on one training cluster running about 200 jobs a day, it detected about 8,550 anomalies. The average detection time was 69 seconds, and the average recovery action time was 94 seconds.

Why it matters

When training stops, GPUs sit idle and time and money are wasted. Automatic recovery can reduce these losses and affect the cost and timeline of AI development.