AWS’s SageMaker AI walkthrough derives autoscaling thresholds for Kokoro-82M from measured request capacity and scale-out time, rather than preset estimates.
AWS published a walkthrough on October 22, 2026, showing how teams can derive autoscaling settings for a text-to-speech endpoint from measurements of the deployed model. The example runs Kokoro-82M, an 82-million-parameter model, in a custom container on an Amazon SageMaker AI real-time endpoint using an ml.g5.xlarge instance.
The method centers on two values: the highest per-instance concurrency that still meets a latency target, and the time required for new capacity to begin serving traffic. In AWS’s test, the endpoint met a 3,000-millisecond, 95th-percentile time-to-first-audio-byte objective at concurrency 50, where latency measured 2,805.7 milliseconds. At concurrency 60, latency rose to 3,374.8 milliseconds, crossing the objective. AWS therefore set the measured ceiling, Q_max, at 50.
A forced scale-out took 260.2 seconds from capacity request to traffic recovery. AWS added a separately bounded 60-second policy-detection allowance, producing a planning figure of 320.2 seconds. Using those measurements and a chosen 1.25x surge ratio, the example derives a scale-out threshold of 37.5 concurrent requests and a scale-in threshold of 25. The sample configuration uses a 30-second scale-out cooldown and a 960-second scale-in cooldown.
The policy uses SageMaker AI’s native ConcurrentRequestsPerModel metric, rather than relying on a custom metric from the container. The example also uses an admission limit of 50 queued or executing requests; beyond that, the container rejects new requests instead of allowing them to wait past the latency target.
AWS cautions that these figures apply to this model and instance configuration, not to every endpoint. The post also flags a pinned PyTorch version with a known security advisory and says it should be reviewed and upgraded before use beyond a proof of value. The practical takeaway is a measurement process that teams can repeat with their own model, hardware, and latency objective.