Search & Indexing at Scale with Elasticsearch

Why Database Queries Were Not Enough

As data volume grew, search and filtering queries became slower and harder to maintain. Complex joins and conditional logic made even simple queries expensive.

User-facing latency increased, and database load became a bottleneck.

Introducing a Dedicated Search Layer

Search concerns were separated from transactional storage. Elasticsearch was introduced as a dedicated read-optimized layer designed specifically for querying and aggregation.

The primary database remained the source of truth, while search became eventually consistent.

Index Design

Indexes were modeled around how users searched, not how data was stored. Denormalized documents reduced query complexity and improved response times.

Asynchronous Indexing

Index updates were handled asynchronously via events. Writes to the primary database never waited on search updates.

This ensured search failures could not impact core business operations.

async function indexTransaction(tx: Transaction) {
  await elastic.index({
    index: 'transactions',
    id: tx.id,
    document: {
      userId: tx.userId,
      amount: tx.amount,
      currency: tx.currency,
      status: tx.status,
      createdAt: tx.createdAt,
    },
  });
}

async function indexTransaction(tx: Transaction) {
  await elastic.index({
    index: 'transactions',
    id: tx.id,
    document: {
      userId: tx.userId,
      amount: tx.amount,
      currency: tx.currency,
      status: tx.status,
      createdAt: tx.createdAt,
    },
  });
}

Performance & Relevance

Search latency dropped significantly, filters became more expressive, and database load was reduced. Result ranking and aggregation enabled better reporting and UX.

Lessons Learned

• Search deserves its own architecture
• Eventual consistency is acceptable for reads
• Query-driven modeling outperforms normalized schemas