Back to news

AI · Official

Amazon SageMaker AI Adds KV Cache-Aware Routing for Real-Time Endpoints

Amazon AWS Machine Learning blog ·

At a glance

Amazon SageMaker AI now offers KV cache-aware routing for real-time endpoints. It sends each request to an instance that already holds a long cached match for the prompt and is not too busy. Teams serving large language models can turn it on with one environment variable and a routing setting, and clients keep using the same APIs.

Key points

Why it matters

It can make large language model apps faster and cheaper to run by reusing work already done on the same prompt, especially for long chats and shared prompts.

Original source

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Amazon AWS Machine Learning blog · Nov 18, 06:11 KST

Read on Amazon AWS Machine Learning blog ↗

AI summarized the original article. Check the original for accuracy. Copyright belongs to Amazon AWS Machine Learning blog.

AI