Red teaming

The word comes from military drills, where the side playing the enemy was called the red team. An AI red team tries to make a model give forbidden answers, reveal hidden information, or lure an agent into dangerous actions, to find where it breaks.

Human experts sometimes do this by hand, and other times another AI writes huge numbers of attack prompts so it can run automatically. The weak points they find are fixed before release. But not finding any in a test does not mean there are none, so people have to keep watching after release.

It is like a bank hiring a skilled locksmith to try to open its vault before it starts using it.

New model announcements often say they "went through red-team evaluation." Looking at who tested which risks, and how much, tells you how seriously the company takes safety.