Using NVIDIA NeMo to Turn AI Agent Failures into Tests and Harness Fixes

At a glance

NVIDIA explains how to analyze the failure records of AI agents to find areas for improvement, turn those findings into tests, and fix the agent's setup. The process decides what to fix first based on the failure records. After fixes, the agent is tested on new and existing tasks to confirm it has improved.

Key points

1. 1

   Repeated problems are found in the execution logs (traces) an agent leaves behind, and they are ranked by priority.

2. 2

   Each problem is linked to the log's unique ID, so it is possible to see exactly where things went wrong.

3. 3

   After fixing the harness (the settings and code that wrap the agent, such as its instruction text), the tests are run again.

4. 4

   In the Dassault Systèmes case, incorrect tool calls dropped from 142 to 0, and the overall score rose 28%.

Why it matters

For companies to use AI agents in real work, general performance scores are not enough. They need ways to find and fix real failures.