AI · Official
NVIDIA Recovery Automatically Restarts Stalled Distributed AI Training Jobs
NVIDIA Technical blog ·
At a glance
Large-scale AI training often stops because of hardware or software problems. NVIDIA Recovery is an open-source service that watches training jobs running on a cluster around the clock. When a stall happens, it finds the cause and handles recovery automatically. It is meant to reduce the burden of people checking logs day and night.
Key points
- The service detects training stalls using application logs, GPU health metrics, and job scheduler signals.
- After detection, it finds and isolates the faulty equipment, then restarts or stops the job from the saved checkpoint (a mid-training save point).
- It can connect to Slurm (a job scheduling program). On first installation, it starts in monitor mode, which only shows recovery recommendations without taking any action.
- From July 9 to October 9, on one training cluster running about 200 jobs a day, it detected about 8,550 anomalies. The average detection time was 69 seconds, and the average recovery action time was 94 seconds.
Why it matters
When training stops, GPUs sit idle and time and money are wasted. Automatic recovery can reduce these losses and affect the cost and timeline of AI development.
Original source
Keep Distributed Training Running with Automated Workload Recovery
NVIDIA Technical blog · Nov 3, 07:43 KST
Read on NVIDIA Technical blog ↗AI summarized the original article. Check the original for accuracy. Copyright belongs to NVIDIA Technical blog.
More from this field
- Anthropic Employee Says Claude Helped Prove Riemann Hypothesis
An Anthropic employee and mathematician, Levent Alpöge, says he used the company's Claude chatbot to create a detailed proof of the Riemann hypothesis over about six weeks. The proof has not yet been checked, and it is over 250,000 lines long in Lean, a computer language used to confirm maths proofs.
BBC News · Nov 21, 08:08 KST - Cole Medin Live-Streams His AI Software Factory Turning Ideas Into Working Software
Cole Medin runs his AI software factory live, taking rough requests from viewers through planning, building, and checking until working software runs on screen. He explains how it works and what he learned building it. The stream also shows a game the system built.
Cole Medin · Nov 21, 08:08 KST - AWS Adds Managed Cluster Extensions for Monitoring and User Management in SageMaker HyperPod Slurm
Amazon SageMaker HyperPod, a service for running Slurm clusters used to build large AI models, now has managed cluster extensions. These let teams manage monitoring and user accounts on running nodes through an API or the console, instead of relying on scripts.
Amazon AWS Machine Learning blog · Nov 21, 08:07 KST - AWS Marketplace Lets AI Agents Buy Software on Their Own
AWS has updated its AWS Marketplace so AI agents can buy software, APIs, data, and content on their own, not just find them. Buyers pay in dollars, and charges are added to their AWS bill.
디지털투데이 · Nov 21, 07:37 KST