AI · Official
Amazon SageMaker AI Adds Cross-Region Inference Routing to Keep Models Running
Amazon AWS Machine Learning blog ·

At a glance
Amazon has launched cross-Region inference routing for Amazon SageMaker AI. It lets an app send model requests to whichever AWS Region has spare capacity, without building its own failover code. The router only works with endpoints that are already running and does not add capacity itself.
Key points
- A new resource called an inference router gives an app one entry point across SageMaker AI endpoints in several AWS Regions and accounts.
- Requests stay in the local Region while its endpoints are healthy and move to another registered endpoint when they are not.
- Health is scored from server errors, throttles, timeouts, and platform failures, while bad input requests do not lower the score.
- Traffic returns gradually after recovery, and CloudWatch shows each destination as healthy, overloaded, or degraded.
Why it matters
Teams no longer need to write and maintain their own cross-Region failover code, which keeps AI services available when one Region runs short of GPUs.
Original source
Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI
Amazon AWS Machine Learning blog · Nov 20, 03:20 KST
Read on Amazon AWS Machine Learning blog ↗AI summarized the original article. Check the original for accuracy. Copyright belongs to Amazon AWS Machine Learning blog.
Other coverage
More from this field
- AI Usage Hits Record Highs as Prices Fall and Growth Cools
US companies are using AI models more than ever, while paying less per unit. Ramp's AI Index, based on transaction data from more than 85,000 US companies, shows token prices fell 21.2 percent in six weeks while token volume hit a record high. Growth in AI company revenue is also slowing over shorter periods.
The Decoder · Nov 20, 04:28 KST - Anthropic Python SDK 2.11.0 Fixes Space Encoding in Query Strings
Anthropic released version 2.11.0 of its Python SDK, a toolkit developers use to connect their code to Claude. The update changes how spaces in web request query strings are written. Now a space is encoded as %20, so AWS SigV4 request signatures match when a parameter contains a space.
anthropics/anthropic-sdk-python 릴리스 · Nov 20, 04:14 KST - Anthropic IPO Could Send $155 Billion to Charity, Report Estimates
Anthropic's stock market debut could send a very large amount of money to charity. Its Public Benefit Fund got 5% of the company's shares, and employees have pledged shares that the company will match. Together, the possible donations add up to about $155 billion.
TechCrunch · Nov 20, 04:10 KST - Accenture Shows How to Cut AI Token Costs on Amazon Bedrock
Accenture used a set of six levers to cut AI token spending on Amazon Bedrock, reporting cost drops of about 40 to 85 percent in measured cases. The article explains how to find where AI money goes and how to apply these levers.
Amazon AWS Machine Learning blog · Nov 20, 03:48 KST