AWS describes deploying llama.cpp models on SageMaker AI and extending the runtime with SemIf to score predefined choices on CPU or GPU.
AWS published a deployment guide on October 12, 2026, showing how to run llama.cpp models on Amazon SageMaker AI for applications that must choose among specified outcomes. The guide covers both standard text generation and a structured-decision option, using SageMaker endpoints to serve models packaged in GGUF format.
In the standard configuration, AWS’s llama.cpp Deep Learning Container accepts OpenAI-compatible chat-completion requests. A model archive stored in Amazon S3 can be served without creating a custom container; the SageMaker endpoint maps invocations to llama.cpp’s chat-completions interface. The guide also describes an alternative extension using SemIf, formerly OpenJev, which returns probabilities and logits for options supplied by the application rather than generating an answer as text.
The structured format is aimed at tasks such as support-queue routing, incident prioritization, policy checks and agent-action selection. Requests supply the situation, a question and an ordered list of allowed options. Applications can use stable option identifiers to apply their own thresholds or fallback rules, but AWS cautions that model outputs should not replace authorization or deterministic business rules. It also recommends testing with labeled decisions from the intended application before production use.
AWS reports a benchmark using 231 public JevBench tasks and a Qwen3.5-4B model quantized as Q4_K_M. The CPU setup recorded 77.5% accuracy, a 1.715-second median client latency and a 19.291-second 95th-percentile latency. The GPU setup recorded 77.9% accuracy, 0.352 seconds at the median and 0.599 seconds at the 95th percentile. AWS says the one-answer accuracy difference should be treated as runtime variation, not evidence that GPU use improves model quality; the measurements include network latency and are not service guarantees.
The guide presents CPU as an option for less time-sensitive work and GPU for interactive decisions or tighter latency targets. Teams can use the basic chat endpoint for generated responses, adding the SemIf adapter when they need direct scoring of declared choices.