Scalable User Search with Amazon Cognito: A Deep-Dive Analysis

Cognito's ListUsers accepts a Filter parameter with proprietary syntax and supports only prefix matching on indexed attributes — email, phone_number, cognito:user_status, and a handful of others. There is no fuzzy search, relevance scoring, custom-attribute search (custom:*), or sorting by arbitrary fields. The default rate limit for ListUsers is 5 RPS per User Pool in most regions, with limited burst. In a financial system with dozens of microservices calling this endpoint — onboarding, KYC, customer support, back-office — that ceiling is hit within seconds.

Beyond throttling, there is a structural latency problem. Each ListUsers call with pagination (PaginationToken) is sequential; you cannot parallelize a scan of a 500,000-user pool. Full-pool iteration can take minutes, making any real-time use case infeasible.

The third failure vector is privacy. Returning user attributes directly from a search API — even an internal one — without field-level filtering exposes PII (name, national ID, date of birth) to consumers that may only need the sub (unique identifier). In environments governed by LGPD or PCI-DSS, this is a concrete compliance risk, not a theoretical one.

The central mechanism is an event-driven asynchronous synchronization pipeline. Cognito exposes Lambda Triggers for lifecycle events: PostConfirmation, PostAuthentication, PreTokenGeneration, and for deletion, a trigger via AdminDeleteUser that can be intercepted with a custom Lambda before the SDK call. Each of these triggers publishes an event to EventBridge with a versioned schema — { "source": "identity.cognito", "detail-type": "user.created" } — containing only the user's sub and the non-sensitive attributes required for indexing.

The sync Lambda consumes these events and performs an idempotent upsert in OpenSearch using the sub as the document _id. Idempotency is not optional here: EventBridge guarantees at-least-once delivery, and a duplicate event without idempotency results in unnecessary re-indexing with version inconsistency risk. The solution is to use OpenSearch's _seq_no and _primary_term fields for optimistic concurrency control, or simply accept that an upsert by _id is naturally idempotent for the search use case.

For the initial load — migrating an existing pool of N users — the pattern is a backfill job that uses ListUsers with pagination, but executed exactly once, off the critical path, with explicit rate limiting (max 4 RPS to stay below the 5 RPS limit) and a DynamoDB checkpoint for resumption on failure. This job does not need to be fast; it needs to be correct and resumable.

The most common mistake I see in search implementations over identity data is indexing everything and filtering at presentation time. This violates LGPD's data minimization principle (Art. 6, III) and creates an index that, if compromised, exposes the full profile of every user. The correct approach is to index only the fields needed for search and return only the fields needed by the consumer.

In OpenSearch, this translates into two distinct design decisions. First, the index mapping must omit sensitive fields such as national ID, date of birth, and full phone number — these fields must not exist in the index. If an attribute is not in the mapping, it cannot be leaked. Second, response projection via _source filtering in the query DSL must be applied by the Search Lambda based on the caller's JWT token scope. A token with scope search:basic receives only { sub, display_name, email_prefix }; a token with scope search:admin receives additional fields such as account_status and created_at.

The sync Lambda must also apply hashing or truncation before indexing. For example, indexing email_domain (the part after @) instead of the full email enables corporate domain searches without exposing individual addresses. For name search, techniques like n-gram tokenization in OpenSearch enable autocomplete without storing the full name in plaintext — though this increases index size by 3-4x and must be evaluated against cluster storage cost.

Silent drift between Cognito and OpenSearch is the most insidious failure mode. If the sync Lambda fails silently — for example, an IAM permission error after a role rotation — the index ages without a visible alarm. The mitigation is a periodic reconciliation job (daily or weekly) that compares counts and random samples between Cognito and OpenSearch, publishing a search.index.drift_count metric to CloudWatch with an alarm on any value > 0 for more than 1 hour.

Index explosion from dynamic custom attributes is another vector. Cognito allows up to 50 custom attributes per User Pool. If the sync Lambda indexes all of them without an explicit mapping, OpenSearch will use dynamic mapping and create fields for each variation, leading to a mapping explosion that can bring down the cluster. The solution is to always define an explicit mapping with dynamic: false and an allowlist of indexable fields.

Cascading throttling during backfill is an operational risk. If the initial backfill and the real-time sync pipeline compete for the same Lambda worker pool, the backfill can exhaust reserved concurrency and delay real lifecycle events. The solution is to run the backfill in a separate Lambda function with isolated reserved concurrency and a dedicated SQS queue with a conservative batch size (10-20 messages).

PII leakage via query logging is frequently overlooked. OpenSearch can log full queries to CloudWatch Logs, including search terms that may contain names or email fragments. In regulated environments, slow logs and audit logs must be configured with field masking or disabled for sensitive fields.

In a financial environment, the user search layer is a high-value target: any data leak here can result in regulatory fines and reputational damage. The security model must follow the principle of least privilege at every hop.

The Search Lambda must have an IAM role with permission only for es:ESHttpGet and es:ESHttpPost on the specific OpenSearch domain ARN, with an aws:SourceVpc condition to ensure calls only occur within the VPC. OpenSearch access must be configured with Fine-Grained Access Control enabled, mapping the Lambda IAM role to an OpenSearch role with read-only permissions on the users-v2 index.

The Sync Lambda needs es:ESHttpPut and es:ESHttpDelete on the same domain, but must use a separate role — never the same role as the Search Lambda. Separating read and write roles limits the blast radius if one of the functions is compromised.

The OpenSearch domain must be deployed inside a private VPC, with no public endpoint, and Security Groups restricting access only to the relevant Lambdas. The KMS CMK for encryption at rest must have a key policy that allows only the Lambda roles and explicitly named administrators — no kms: for Principal: "".

For auditing, CloudTrail must be enabled with data events for the OpenSearch domain, and the API Gateway Access Log must be sent to an S3 bucket with Object Lock enabled (COMPLIANCE mode, 90-day retention for PCI-DSS) to guarantee immutability of access logs.