Hybrid Lexical-Semantic Search

Lexical search works well for exact product queries — “iPhone 15 128GB black” — but fails when users describe intent: “warm winter jacket under 1000.” At Snapdeal’s scale, bridging this gap required integrating dense retrieval into the existing BM25 pipeline without disrupting latency budgets or indexing workflows.

Architecture

The system layers semantic retrieval on top of the existing Elasticsearch lexical pipeline:

Lexical Baseline (BM25)

The production search stack uses Elasticsearch with custom analyzers tuned for product catalogs — handling Hindi transliteration, brand name normalization, and category synonyms. This path handles exact-match queries with sub-50ms latency.

Dense Retrieval Layer

Product titles and descriptions are encoded into dense vectors using a sentence-transformer model fine-tuned on Snapdeal’s query-click logs. Query embeddings are computed at request time and matched against the product index using approximate nearest neighbor search.

The key engineering challenge was keeping the dense index synchronized with the catalog — products are added, removed, and updated continuously. A streaming pipeline processes catalog changes and updates embeddings incrementally rather than rebuilding nightly.

Fusion and Reranking

Results from both retrieval paths are merged using reciprocal rank fusion (RRF). The fused candidate set passes through a lightweight cross-encoder for final reranking. This two-stage approach keeps the cross-encoder’s latency impact manageable — it only scores the top candidates, not the full corpus.

Evaluation

Offline evaluation used a held-out set of query-click pairs with manual relevance judgments for the top positions.

Metric	Before (BM25 only)	After (Hybrid)
MRR	0.39	0.63
NDCG	0.48	0.72

The largest gains came from “intent” queries where lexical matching fails — queries describing use cases, attributes, or categories rather than specific product names. For exact-match queries, performance was unchanged since the BM25 path still dominates the fusion score.

Deployment

The dense retrieval layer runs as a sidecar service alongside the existing search stack. This architecture let us A/B test the hybrid approach against the lexical baseline without modifying the core search service, and roll back instantly if needed.