← All news
Global · AI & Technology
Global·AI & Technology

Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints

Amazon SageMaker AI now routes requests using reported cache contents and instance load. The feature targets LLM workloads with multi-turn conversations, shared prompts and heavy traffic, while keeping client APIs unchanged.

AI Hubs United Editorial ·

Amazon SageMaker AI has introduced KV cache-aware routing for real-time endpoints, according to the AWS Machine Learning Blog. The feature directs requests toward instances that have already cached matching prompt content, while also accounting for how busy those instances are. It is intended for teams serving large language models in applications such as chat assistants, retrieval-augmented generation and agents with lengthy system prompts.

Unlike prefix-aware routing, which estimates cache placement from request text, the new approach uses cache updates reported by model containers. SageMaker maintains an index that tracks additions and evictions, allowing it to match growing conversation histories against cached tokens. AWS says the index contains hashed pointers rather than prompts, request payloads or cached tensors. If a matching instance is overloaded, the router considers another candidate; without a useful available match, it falls back to load-aware routing.

Teams enable the feature through a model environment variable and an endpoint routing setting, without changing their client APIs. In practical terms, the approach helps requests reuse previous prompt processing rather than repeat work after reaching an instance without the relevant cache. AWS notes that newly added or replaced instances begin with empty caches, which can temporarily increase latency.

AI Hubs United original short summary of the source linked above. It is not the original article; read the full story at the source.