AWS described a nine-stage AgentCore evaluation system for testing agent quality, safety, latency and cost across development and live use.
AWS published a nine-stage evaluation framework for its internal, multi-agent sales intelligence system on October 16, 2026. Built with Amazon Bedrock AgentCore and other AWS services, the system helps field sales teams and leaders query business information through Amazon Quick. AWS says it developed the framework to find incorrect but fluent answers that ordinary infrastructure monitoring may not catch.
The post describes errors found before launch, including a customer’s three-month revenue of $410.08 being presented as $410.08 million, and an annual recurring revenue estimate that was not in the data. In a separate pre-production test, a mismatch between coordinated data, tool and prompt changes reduced the pass rate from 78% to 38%, according to AWS.
The framework evaluates safety, answer quality, performance, cost and operational control at three levels: individual tools, sub-agents and the full multi-agent workflow. Its nine stages cover defining success measures, comparing changes, checks during code integration and deployment, pre-release canaries, adversarial testing, live quality drift, monitoring and feedback. AWS says its evaluators combine AgentCore’s built-in options with custom checks for specific errors, third-party tools and deterministic comparisons against known answers.
AWS also describes production safeguards and monitoring. Bedrock Guardrails screen every interaction at input and output, while online evaluation samples 10% of live traffic. In one example, a canary detected a drop in retrieval quality within a 30-minute window, even though latency and error rates remained normal. The team says automated evaluation now catches quality regressions in under four hours, compared with days or weeks previously, and that prompt iteration cycles fell from days to minutes.
The framework reflects AWS’s experience with its own sales agents; the post notes that other organizations would need to define their own failure categories and evaluation datasets. Its central operational point is to keep measuring agent behavior across tools and workflows after deployment, not rely on response availability as evidence of correctness.