AI · Official
Amazon SageMaker AI Adds KV Cache-Aware Routing for Real-Time Endpoints
Amazon AWS Machine Learning blog ·

At a glance
Amazon SageMaker AI now offers KV cache-aware routing for real-time endpoints. It sends each request to an instance that already holds a long cached match for the prompt and is not too busy. Teams serving large language models can turn it on with one environment variable and a routing setting, and clients keep using the same APIs.
Key points
- KV-aware routing is a new routing strategy for SageMaker AI real-time endpoints.
- The model container reports its cache contents, and SageMaker AI keeps an index of that state.
- Requests go to the instance that holds the longest matching cached part of the prompt, unless it is too busy.
- It is set up with one environment variable in the model and the KV_AWARE setting in the endpoint configuration.
Why it matters
It can make large language model apps faster and cheaper to run by reusing work already done on the same prompt, especially for long chats and shared prompts.
Original source
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
Amazon AWS Machine Learning blog · Nov 18, 06:11 KST
Read on Amazon AWS Machine Learning blog ↗AI summarized the original article. Check the original for accuracy. Copyright belongs to Amazon AWS Machine Learning blog.
More from this field
- AWS: Use code for fixed rules and models only where judgment is needed
AWS's machine learning blog argues that teams should first ask whether a workflow step needs an AI model at all. Many steps, such as input checks, formatting, and simple routing, can be handled by plain code, which saves money and time.
Amazon AWS Machine Learning blog · Nov 18, 06:14 KST - Microsoft Releases Decision 1.1, Claiming 10x Speed and 1/200th the Cost of LLMs
Microsoft has officially launched Microsoft Decision 1.1, a model specialized in judgment tasks. The company says it works faster and costs less than using large language models for the same repetitive decisions. It was built on Google's Gemma 4 26B A4B and adds image analysis.
지디넷코리아 · Nov 18, 06:09 KST - Meta's AI Chief Posts Deepfake Video of Sam Altman on Instagram
Meta's chief AI officer, Alexandr Wang, posted an AI-generated video showing OpenAI CEO Sam Altman and Meta CEO Mark Zuckerberg on a video call. The clip promotes Meta's AI agent Muse, and Meta says it is allowed because it is labeled as AI. Critics say the platform still allows many deepfake images of real people.
The Verge · Nov 18, 06:00 KST - Google Shows Five Projects Built With Gemini 4 Carbon
Google's new Gemini 4 Carbon model is being used by developers for complex projects, such as physics simulations, data tools, and 3D designs. Google shared five examples built by independent creators. The model is available through Google's developer tools and business platforms.
Google Gemini products · Nov 18, 06:00 KST