Design Doc: Continuous Evaluation Suite for Agents with Bedrock AgentCore

LLM agents are not deterministic functions. They combine a language model (which can be updated by the provider without explicit notice), a set of tools with their own APIs and schemas, short- and long-term memory, and an orchestration layer that interprets intent and chains calls. Each of these components can change independently — and the interaction between changes is where the most dangerous regressions hide.

Consider a concrete scenario: a fintech's customer support agent uses a get_account_balance tool that returns values in cents. The backend team migrates the API to return values in the base currency unit without updating the tool documentation. The agent keeps working — it calls the tool, receives a number, and responds to the user. But now it reports balances 100x larger than the real value. No runtime error. No latency alarm. The degradation is semantic, not technical.

This is the central problem a continuous evaluation suite needs to solve: detecting semantic and behavioral degradation before it reaches the user, in a system where parts change asynchronously and errors are rarely exceptions — they are plausible but incorrect responses.

The state of the art in agent evaluation is still consolidating. Most teams I encounter operate with a combination of: manual ad hoc testing before releases, some user satisfaction metrics collected reactively, and latency/error alarms that only capture technical failures. This is insufficient for systems that make decisions with real consequences. What we need is a discipline analogous to what we have in traditional software systems — unit tests, integration, regression, coverage — but adapted to the probabilistic and compositional nature of agents.

The architecture I propose organizes evaluation into three complementary layers, each with distinct purpose, frequency, and cost. The useful metaphor here is defense in depth — no single layer is sufficient, but together they form a robust detection network.

Layer 1 — Offline Evaluation (Pre-Deploy Gate)

Before any agent version is promoted to production, it runs through an offline test battery against a versioned dataset stored in S3. The dataset is structured as a set of test cases, each containing: the user input (or turn sequence for multi-turn conversations), the expected set of tool calls (with expected arguments and tolerances), the expected response (or evaluation criteria when there is no deterministic response), and category metadata (task type, difficulty level, whether it is adversarial).

Bedrock AgentCore runs the agent in evaluation mode — with the same tools configured, but in an isolated environment with mocked tools that return controlled responses. This is critical: you do not want offline tests making real calls to production APIs, but you also do not want mocks so simplified they do not test the agent's reasoning about tool responses.

Metrics collected in this layer include: tool-call accuracy (did the agent call the right tool?), argument fidelity (were arguments correct within tolerance?), call sequence correctness (for tasks requiring a specific call sequence), unnecessary tool calls (did the agent call unnecessary tools, indicating confused reasoning?), and final response quality evaluated by an LLM-as-judge configured via Bedrock Evaluations.

The quality gate is defined as a set of thresholds per metric category. Initial proposal: tool-call accuracy ≥ 0.90 on the full set, ≥ 0.95 on the subset of critical cases (flagged in the dataset), and no regression > 5 percentage points relative to the previous version's baseline. That last criterion — relative regression — is as important as the absolute threshold: an agent that was already poor at something should not get even worse.

Layer 2 — Adversarial Testing

Separate from the standard evaluation dataset, we maintain an adversarial dataset with specific categories: prompt injection attempts (attempts to make the agent ignore its system instructions), boundary probing (inputs that test policy limits — what does the agent do when it receives an ambiguous request that could be legitimate or malicious?), and tool-use edge cases (what happens when a tool returns an unexpected error? Does the agent recover gracefully or loop?).

These tests do not follow the same pass/fail regime as functional tests — they have expected behavior criteria. For prompt injection, the criterion is that the agent maintains its system instructions. For boundary cases, the criterion is that the agent escalates or refuses appropriately rather than hallucinating a response. For tool errors, the criterion is recovery in at most N attempts with graceful fallback.

Layer 3 — Online Signals (Production Monitoring)

In production, we instrument an asynchronous evaluation pipeline that processes a sample of real conversations. Sampling is stratified — we ensure representation of different task types, not just volume. Selected conversations are sent to a Step Functions workflow that: extracts tool-use traces from AgentCore, runs LLM-as-judge evaluation on final turns, and persists metrics to CloudWatch with dimensions by agent version, task type, and tool.

Beyond sampling, we monitor proxy signals that do not require LLM evaluation: rate of conversations hitting the maximum turn limit (indicates agent stuck in a loop), rate of unrecovered tool errors, escalation-to-human rate (if applicable), and latency distribution per tool (latency changes can indicate call behavior changes).

One of the most neglected aspects of agent evaluation systems is management of the evaluation dataset itself. Teams build an initial set of test cases, run them for months, and never revisit — while the agent evolves to handle entirely new use cases that are not represented. The dataset ages and loses discriminative power.

The proposal here is to treat the evaluation dataset with the same discipline we apply to code: explicit versioning, changelog, and a continuous curation process.

Versioning structure: Each dataset version is an immutable artifact in S3, identified by a content hash and a semantic version number (major.minor.patch). DynamoDB maintains the registry with metadata: creation date, author, changelog, coverage by task category, and the agent version it was validated against. The quality gate always runs against the dataset version specified in the agent manifest — not against "latest". This is deliberate: it guarantees reproducibility and prevents a new dataset from breaking an agent that was working well.

Curation process: New test cases enter the dataset through three paths. The first is manual curation — engineers or domain experts write cases based on requirements. The second is production mining — real conversations that were flagged as problematic (via user feedback, escalation, or automatic detection) are reviewed and converted into test cases. This is the most valuable path: these are real failures, not hypothetical ones. The third is synthetic generation — using an LLM to generate variations of existing cases, especially to increase coverage of edge cases and adversarial scenarios.

The dataset drift problem: As the agent improves, cases that were once difficult become trivial. A dataset that is not updated loses discriminative power — the agent passes everything, but that does not mean it is working well on new cases. The solution is to monitor the distribution of dataset scores over time: if most cases score near 1.0, it is time to add harder cases. This is analogous to the benchmark saturation problem in ML — when a model reaches human performance on a benchmark, the benchmark needs to be replaced, not the model declared perfect.

Tool-use coverage: For agents with multiple tools, the dataset must ensure adequate coverage of each tool and tool combinations. We maintain a coverage map showing, for each tool, how many test cases exercise it, in what contexts, and with what argument patterns. New tools added to the agent must have minimum coverage before the agent can be promoted — this is an additional quality gate rule.