AWS demonstrates serving Qwen3.5-9B across two EKS GPU nodes with Ray LLM containers, separating prompt processing from token generation.
AWS outlined a multi-node inference deployment on October 12, 2026, using its Ray LLM Deep Learning Containers on Amazon Elastic Kubernetes Service. The example runs Qwen3.5-9B across two GPU worker nodes and presents the model through a single OpenAI-compatible endpoint.
The configuration splits inference into prefill and decode phases. One GPU node processes the input prompt and builds the key-value cache; the other generates response tokens using that cache. AWS says separating these phases can prevent compute-intensive prompt processing from disrupting token generation, while allowing the two stages to be configured or scaled independently. The approach is presented for workloads including retrieval-augmented generation, document summarization and agent applications.
The sample cluster uses a CPU-based m7i.xlarge system node for the Ray head and cluster services, plus two g6.8xlarge GPU workers. The GPU nodes are placed in the same Availability Zone, since Elastic Fabric Adapter does not operate across Availability Zones. NIXL transfers the cache between workers, with AWS’s Elastic Fabric Adapter stack providing the node-to-node connection. The container bundles components including vLLM, Ray Serve LLM and NIXL, so the deployment is configured through a YAML application manifest rather than a custom serving image.
AWS’s example sets a 4,096-token maximum model length and uses bfloat16 precision. In the documented setup, loading Qwen3.5-9B weights on a restart drops from around 145 seconds to about 2 seconds when the shared-memory and page-cache configuration is used. That figure refers to model loading in this particular configuration, not a general inference-latency result.
The walkthrough also calls out an operational detail: NIXL can fall back to a TCP-based transport if libfabric is unavailable, without producing an error. AWS recommends checking worker logs to confirm that the LIBFABRIC backend is active. The same container approach can also support other multi-node patterns, including tensor, pipeline and data parallelism.