Agentic RAG on AWS: Architecture Bake-Off for Financial-Grade Platforms

Classic RAG — embed, retrieve, generate — solves the static knowledge problem but breaks along three critical dimensions in financial environments. First, multi-hop reasoning: a query like "what is customer X's consolidated credit exposure considering derivatives and open positions as of T-2?" requires multiple retrieval steps with dependencies between them, not a single vector search. Second, tool use: computing VaR, querying a credit limit system, or triggering a real-time pricing API are actions, not documents — and naive RAG has no action mechanism. Third, regulatory traceability: under BACEN, CVM, or SEC, the reasoning chain that led to an automated recommendation or decision must be reconstructible, not just the final output.

Agentic RAG solves all three by introducing a plan-execute-observe loop (ReAct, MRKL, or variants) where the agent dynamically decides which tools to invoke, in what order, and when the answer is good enough to terminate. The cost of that capability is operational complexity: agent loops introduce non-determinism, variable latency, infinite-loop risk, and an expanded attack surface. The central architectural choice is where that loop lives and who controls it — and that is precisely where the four approaches diverge materially.

Option A — Native Bedrock Agents: The agent lives entirely inside Bedrock. AWS manages the ReAct loop, tool routing (Action Groups via Lambda), session memory, and Knowledge Base integration (OpenSearch Serverless as the vector backend). The operator defines the instruction prompt, tool definitions in OpenAPI schema, and guardrails. Orchestration latency runs around 800ms–1.5s per reasoning step for Claude 3 Sonnet, billed per input/output token plus orchestration overhead.

Option B — LangGraph + EKS: The agent loop runs in Python inside EKS pods, using LangGraph to define the state graph. The team has full control over every graph node, transitions, state checkpointing in DynamoDB, and integration with any retrieval backend — OpenSearch Service (provisioned), pgvector on Aurora, or Kendra. Orchestration latency is deterministic and controllable, but operational responsibility is total.

Option C — Step Functions-orchestrated: The agent loop is externalized to a Step Functions Express Workflow. Each reasoning step is a machine state — invoke model, evaluate, branch, retrieve, tool call. LLM non-determinism is contained within individual states; the orchestration itself is deterministic, auditable in X-Ray, and replayable. Lambda executes the tools; Bedrock or SageMaker serves the model.

Option D — Hybrid Bedrock + Step Functions: Bedrock Agents handles the inner ReAct loop while Step Functions orchestrates the outer business flow — input validation, context enrichment, agent invocation, post-processing, audit logging. The agent is a controlled black box, not the primary orchestrator.

Regulatory auditability is where Step Functions (Option C) has a structural advantage. Each machine state generates an EventBridge event and an X-Ray entry with the full payload — input, output, duration, errors. In a BACEN audit, I can reconstruct exactly what the system decided at each reasoning step for any historical session. Bedrock Agents (Option A) provides CloudTrail for API calls and invocation logs in CloudWatch, but the internal reasoning loop is opaque — you see the input and final output, not the intermediate thinking steps, unless you explicitly enable trace via enableTrace: true in the InvokeAgent call.

Inference cost is the second differentiator. In Bedrock Agents, each reasoning step consumes tokens from the system prompt + conversation history + retrieved context + response. For Claude 3 Sonnet (us-east-1), that means roughly $3/1M input tokens and $15/1M output tokens. An agent with 5 reasoning steps and average context of 4k tokens per step costs ~$0.08 per session. In Step Functions Express, orchestration cost is $1/1M state transitions — practically zero — but you still pay for Bedrock tokens. The real difference is that Step Functions lets you terminate the loop early with deterministic evaluation logic, reducing unnecessary steps.

P99 latency is where LangGraph+EKS (Option B) can surprise negatively. Pod cold starts, state serialization overhead in DynamoDB, and network latency to provisioned OpenSearch Service can push P99 to 8–12 seconds under complex workloads. Bedrock Agents, being managed serverless, has more consistent P99 around 4–7 seconds for 5 steps, but without tuning control.

In regulated financial environments, the attack surface of a RAG agent is qualitatively different from a REST API. The agent can be induced via prompt injection to exfiltrate context data, invoke unauthorized tools, or bypass guardrails. Each architecture has a distinct risk profile.

In Bedrock Agents, native Guardrails offers configurable content filters with harm categories (HATE, INSULTS, SEXUAL, VIOLENCE, MISCONDUCT, PROMPT_ATTACK) and word filters with custom lists — configurable via the CreateGuardrail API with contentPolicyConfig and wordPolicyConfig. The problem is that Guardrails evaluates the output, not the intermediate reasoning. An injection that manipulates the planning step can go undetected if the final output appears benign.

In LangGraph+EKS, security responsibility is entirely the team's. That means implementing: (1) input sanitization before any model call, (2) IAM roles with least-privilege per graph node — a retrieval node should not have permission to invoke payment APIs, (3) KMS CMK for state encryption in DynamoDB with aws:kms encryption type and per-tenant key in multi-tenant environments, (4) VPC endpoints for OpenSearch and Bedrock eliminating public internet traffic.

In Step Functions, the state separation creates a natural opportunity to insert deterministic validation between steps — a ValidateToolOutput state that checks schema, range, and permissions before passing the result to the next reasoning step. This is difficult to do reliably in Bedrock Agents without extensive Action Group customization. For environments with LGPD/GDPR requirements, Step Functions also facilitates data residency controls via IAM condition aws:RequestedRegion on each service invocation.

Agentic RAG breaks traditional API SLOs because latency is a function of the number of reasoning steps, which is non-deterministic. Defining a P99 < 5s SLO for an agent with up to 8 steps is mathematically impossible without explicit termination control.

For Bedrock Agents, the primary observability signals are: InvokeAgent duration in CloudWatch (metric InvocationLatency), step count via trace parsing (field orchestrationTrace.rationale), and THROTTLING_EXCEPTION rate indicating pressure on the account TPS limit (default: 10 TPS per model per region, increasable via Service Quotas). A realistic SLO is P95 < 8s with a 0.5% error budget for throttling.

For Step Functions, each state emits native metrics: ExecutionTime, ExecutionsFailed, ExecutionsTimedOut. With OpenTelemetry, I can instrument each tool's Lambda with spans that propagate the Step Functions traceId, creating an end-to-end trace tree in X-Ray or Datadog. The most useful SLO here is steps per session — if P95 > 6 steps, that's a signal that the system prompt or retrieved documents are low quality, not that the system is slow.

For LangGraph+EKS, the pattern that works is the OpenTelemetry SDK with LangChain auto-instrumentation, exporting to ADOT Collector on EKS and then to CloudWatch EMF or Datadog. Critical metrics: llm.token.usage per graph node (for per-step cost control), retrieval.hit_rate (relevant documents / total retrieved), and tool.error_rate per tool name. A retrieval.hit_rate < 0.6 in production is an alert for vector index degradation — likely embedding drift or a stale index.