Why LLM Serving Settings That Win on Paper Can Fail in Production

At a glance

This video introduces an optimization agent DigitalOcean runs on its AMD MI355 inference servers. The agent proposes and applies settings for vLLM (serving software that helps LLMs respond quickly) and Kubernetes (a tool for managing server clusters), then measures throughput under real latency targets.

What the video covers

1. 1

   After applying a configuration, the agent measures throughput under latency targets, keeps better settings, and safely restores the old setup otherwise.

2. 2

   Four studies are shared, including a +4% gain that turned out to be run-to-run noise.

3. 3

   The same agent is being extended to disaggregated serving, which separates prefill (input processing) and decode (answer generation).

Why watch it?

It is useful for engineers who want faster LLM services or who run inference infrastructure, because it shows a measurement method closer to what real users experience.