ADR: OpenSearch Serverless vs Dedicated Vector Database for Agentic RAG

An agentic RAG platform differs from a simple RAG chatbot in two critical dimensions: autonomy and dynamic scope. Agents don't just retrieve documents — they decide when to retrieve, which collections to query, and how to combine multiple sources to build multi-hop reasoning. This imposes requirements that a static vector index alone cannot satisfy.

The first set of forces comes from the multi-tenant model. Each tenant has its own knowledge base — legal documents, technical manuals, internal policies — and cannot, under any circumstances, see data from another tenant. This is not just logical isolation via metadata filter; it is a security requirement that must be auditable and demonstrable. The naive approach of putting all vectors in a single index and filtering by tenant_id creates residual risk: a malformed query or a filter bug exposes cross-tenant data. Separate indexes per tenant eliminate this risk, but explode operational costs in provisioned vector databases.

The second set of forces is economic. Dedicated vector databases like Pinecone operate on a provisioned pod model. For 50 tenants with heterogeneous volumes — some with 100k vectors, others with 50M — over-provisioning is inevitable. The cost of keeping p2 or p3 pods idle during low-demand hours is real and recurring. OpenSearch Serverless, by contrast, charges per OCU (OpenSearch Compute Unit) actually consumed, with automatic scaling to near-zero in periods without traffic — which for workloads with a strong daytime pattern represents a structural saving.

The third set of forces is Bedrock ecosystem integration. AWS announced native support for OpenSearch Serverless as a Bedrock Knowledge Bases backend, meaning the ingestion pipeline (chunking, embedding with Titan Embeddings v2, indexing) is managed by the service itself. This eliminates an entire layer of orchestration code I would otherwise have to write and maintain if I chose an external vector database.

The most underestimated aspect in choosing a vector engine for enterprise RAG is the permission filter at query time — what the literature calls pre-filtering vs post-filtering. The distinction matters far more than it appears.

Post-filtering is simple to implement: you retrieve the top-K most similar vectors and then discard those the user doesn't have permission to see. The problem is that if 60% of the most semantically relevant documents are outside the user's permission scope, the final result can have catastrophically low recall — you return 2 documents when you should return 10, and the 2 you returned are not the best ones.

Pre-filtering solves this by applying the filter before the vector search, restricting the search space to only the documents the user can see. OpenSearch Serverless supports pre-filtering via filter in the context of knn_vector queries using the _search API with knn + filter combined. This is critical: it means the agent never sees a document the user shouldn't see, and semantic relevance is calculated within the correct permission space.

Additionally, the platform needs to support incremental ingestion — new documents continuously arrive via S3 events, and the index needs to reflect this in minutes, not hours. OpenSearch Serverless handles this natively: Bedrock Knowledge Bases has incremental sync support via S3 data source, where only modified or added chunks are re-embedded and re-indexed. This contrasts with solutions like pgvector on RDS, where high-frequency incremental ingestion creates write IOPS pressure and can degrade read performance without careful tuning of vacuum and HNSW indexes.

A point many architects overlook: hybrid search (vector + BM25 lexical) is not optional for quality RAG in technical domains. Documents with serial numbers, product codes, technical acronyms, or specific proper nouns are poorly retrieved by purely semantic search. OpenSearch supports hybrid search natively with hybrid query type, combining kNN and BM25 scores via min-max normalization or RRF (Reciprocal Rank Fusion). Pinecone requires you to maintain a separate text index (typically Elasticsearch or OpenSearch) and do the fusion at the application layer — adding latency, operational complexity, and an extra point of failure.

The decision to use one AOSS collection per tenant deserves detailed justification, because it goes against cost-optimization intuition at first glance.

The obvious alternative would be a single shared index with a tenant_id field as a filter. This works in low-risk environments, but fails in three real scenarios: (1) compliance auditing — when an auditor asks "how do you guarantee Tenant A doesn't see Tenant B's data?", the answer "we have a filter in the query" is not satisfactory; the answer "each tenant has its own collection with isolated IAM data access policy" is; (2) maintenance operation blast radius — a reindex or schema migration on a shared index affects all tenants simultaneously; (3) performance isolation — a tenant with massive ingestion on a shared index can degrade query latency for other tenants via indexing resource contention.

Per-tenant collection resolves all three. The minimum cost of ~2 OCUs per active collection is real, but for paying tenants on B2B platforms, this cost is absorbed in the plan pricing. For free-tier or very low-volume tenants (less than 50k vectors, less than 100 queries/day), the strategy is different: these tenants are grouped into a shared free-tier collection, where isolation is done by metadata (tenant_id as a mandatory filter field in all queries). The trade-off of isolation guarantee for cost is explicit and documented in the free-tier service level agreement.

This two-layer strategy — dedicated collections for paying tenants, shared collection for free-tier — is a pattern I consistently use in multi-tenant platforms: you don't need a single solution that serves all cases, you need a clear policy that defines which solution applies to which segment.