AWS reports that bi-directional KV-cache transfer in vLLM lowered time to first token by up to 1.6× and 3.3× in tests.
AWS published details on October 19, 2026, about using bi-directional key-value (KV) cache transfer with vLLM to reduce time to first token (TTFT) in multi-turn agent workloads. The approach lets prefill and decode servers reuse cached conversation context across turns instead of repeatedly calculating it.
In the conventional setup described by AWS, the prefill server sends KV data to the decode server, but not back again. After a response, the decode server holds cached representations of both the prompt and generated tokens. On a later turn, the prefill server may have to recompute that context—especially if its local cache has been cleared or the request goes to another prefill server. That repeated work can raise latency as conversations grow.
AWS tested the method in a one-prefill, one-decode deployment on Amazon EC2 P5en instances connected with Elastic Fabric Adapter. With Qwen3-32B, the company measured up to a 1.6× TTFT reduction; with zai-org/GLM-5-FP8, it reported up to 3.3×. The benchmarks disabled prefill-side prefix caching to simulate cache misses, and AWS says results vary by workload and configuration.
The implementation returns eligible cached blocks from decode to prefill, coordinated by a router that tracks cache-location metadata for each conversation. AWS says the router stores metadata rather than KV tensors. Decode-side blocks have a configurable retention period, set to 480 seconds by default, while a recompute threshold—64 tokens by default—determines when fetching cached data is preferable to recalculating it.
AWS says the feature is available in vLLM version 0.21.0 or later with NixlConnector, with bi-directional transfer enabled on both server types and a KV-cache-aware router. The results are specific to the tested setup, not a guarantee of the same improvement across deployments.