Back to news

AI · Official

NVIDIA Recovery Automatically Restarts Stalled Distributed AI Training Jobs

NVIDIA Technical blog ·

At a glance

Large-scale AI training often stops because of hardware or software problems. NVIDIA Recovery is an open-source service that watches training jobs running on a cluster around the clock. When a stall happens, it finds the cause and handles recovery automatically. It is meant to reduce the burden of people checking logs day and night.

Key points

Why it matters

When training stops, GPUs sit idle and time and money are wasted. Automatic recovery can reduce these losses and affect the cost and timeline of AI development.

Original source

Keep Distributed Training Running with Automated Workload Recovery

NVIDIA Technical blog · Nov 3, 07:43 KST

Read on NVIDIA Technical blog ↗

AI summarized the original article. Check the original for accuracy. Copyright belongs to NVIDIA Technical blog.

AI