Summary
Aftermath provides labeled agent traces and a command-line evaluation kit for assessing outcome-based guardrails.
About
Aftermath is a focused testing and evidence-generation tool for teams evaluating whether AI-agent guardrails detect harmful outcomes that tool-name allowlists may miss. Its small labeled corpus offers a concrete starting point for agent safety assessment, but the stated classification results apply only to that corpus.
Aftermath is an evaluation kit for testing agent guardrails using labeled outcomes rather than tool names. It includes 30 agent traces covering cases such as write-capable HTTP GET requests, data exfiltration, proxy behavior, test-environment escapes, and resource abuse. Each trace is labeled as safe, data-exfil, unauthorized-write, scope-escape, or resource-abuse. The product reports 30/30 classifications for its outcome classifier on this corpus, compared with 8/30 for a tool-name allowlist. It includes a standard-library command-line interface, a runbook, and a one-page evidence report intended to support the NY RAISE Act 72-hour reporting clock, which begins January 1, 2027.
Key Features
A corpus of 30 labeled agent traces covering several guardrail failure scenarios
Outcome labels for safe behavior, data exfiltration, unauthorized writes, scope escapes, and resource abuse
An outcome-classifier evaluation reporting 30/30 on the supplied corpus
A comparison with a tool-name allowlist reporting 8/30 on the same corpus
A standard-library CLI, runbook, and one-page evidence report
Real-World Use Cases
Agent guardrail evaluation
Use the labeled traces to test whether an agent safety classifier identifies risky outcomes that a tool-name allowlist may not detect.
Safety testing for tool-using agents
Assess agent behavior in scenarios involving data exfiltration, unauthorized writes, scope escapes, and resource abuse.
Incident evidence preparation
Use the included runbook and evidence report as inputs to internal incident documentation and reporting workflows.
Business Opportunities
Agent safety evaluation service
Problem: Organizations deploying tool-using AI agents may lack practical tests for harmful outcomes that are not apparent from tool names alone.
Solution: Use Aftermath's labeled traces and CLI as a baseline evaluation, then adapt testing to the organization's agents and risk scenarios.
Estimated ROI: Potentially moderate through earlier detection of agent safety gaps and reduced manual test-case preparation; actual returns depend on deployment scale and the cost of incidents.
Execution: Run the supplied corpus against existing guardrails, document results, and identify gaps requiring additional organization-specific tests. Treat the reported corpus score as a limited benchmark rather than a general performance guarantee.
Compliance evidence workflow integration
Problem: Teams may need a repeatable process for documenting AI-agent incidents and preparing evidence within applicable reporting timelines.
Solution: Incorporate the included runbook and one-page evidence report into internal incident-response procedures, with legal and compliance review.
Estimated ROI: Potentially moderate through more consistent evidence preparation and reduced process friction; the tool alone does not establish regulatory compliance.
Execution: Map the report and runbook to existing incident workflows, assign review responsibilities, and validate that the materials meet the organization's legal and regulatory requirements.