Engineering Atlas › System Design Interviews

Design a semantic search service

19 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope: answer generation, model training, web crawling, personalisation.

Back-of-envelope

All prices and throughputs below are labelled assumptions. Replace them with your vendor's current numbers.

API

POST /v1/search
{
  "tenant": "acme",
  "query": "why does checkout return E1234 after the card step",
  "filters": { "source": ["wiki", "tickets"], "updated_after": "2026-01-01", "lang": "en" },
  "mode": "hybrid",                  // "vector" | "keyword" | "hybrid"
  "k": 10,
  "rerank": true,
  "user": "u_42"                     // used to build the ACL filter server side
}

200 OK
{
  "results": [
    { "chunk_id": "d9f1:0007", "doc_id": "d9f1", "score": 0.83,
      "snippet": "...E1234 is raised when the 3-D Secure step times out...",
      "source": "tickets", "updated_at": "2026-05-02T10:14:00Z" }
  ],
  "index_version": "emb-v3@2026-09-30",
  "stages": { "embed_ms": 14, "retrieve_ms": 38, "rerank_ms": 61 }
}
PUT    /v1/documents/{doc_id}     // upsert; body: text or blob ref, metadata, ACL
DELETE /v1/documents/{doc_id}     // tombstone now, physical delete later
POST   /v1/reindex                // admin: build a new index version for a new model

The caller never sends the ACL. The service derives the allowed groups from user through the identity system, because a client-supplied filter is a client-controlled security boundary. The response reports index_version so that an evaluation run can tell which model and snapshot answered.

Data model

Chunk record, the unit that is indexed and returned:

FieldPurpose
chunk_iddoc_id:ordinal, stable across re-embedding of the same text
doc_id, ordinal, char_start, char_endLink back to the source, and neighbour expansion at read time
textStored for snippets, BM25 and re-ranking input
embedding, model_idThe vector and the model version that made it
tenant, source, lang, updated_atFilterable metadata, each with an index
acl_groups[]Group ids allowed to read, copied from the source system
content_hashHash of chunk text, to skip re-embedding unchanged chunks
deletedTombstone flag until the next merge

Partitioning. Shard by hash(doc_id), so all chunks of a document live together and an edit touches one shard. Tenant is a filter, not the shard key; a very large tenant gets dedicated shards.

Indexes per shard: an HNSW graph (or IVF lists) over embedding, an inverted index with BM25 statistics over text, and secondary indexes on the filter fields. The ACL filter is "chunk groups intersect user groups", a posting-list union.

Source of truth. Documents live in object storage; the index is derived and rebuildable.

High-level design

flowchart LR
  SRC["Source systems"] --> CDC["Change feed"]
  CDC --> Q["Ingest queue"]
  Q --> PARSE["Parse + chunk"]
  PARSE --> EMB["Embedding workers (GPU)"]
  EMB --> WR["Index writers"]
  WR --> IDX["Search shards: HNSW + BM25 + filters"]
  WR --> DELTA["Fresh delta layer"]
  U["Client"] --> API["Search API"]
  API --> QE["Query embedder"]
  API --> ACL["ACL resolver"]
  QE --> RT["Shard router"]
  ACL --> RT
  RT --> IDX
  RT --> DELTA
  IDX --> FUSE["Fusion (RRF)"]
  DELTA --> FUSE
  FUSE --> RR["Re-ranker"]
  RR --> API

Write path. A change feed (webhooks, polling by updated_at, or database change capture) puts a document event on a queue, keyed by doc_id so events for one document stay in order. A parser extracts text, a chunker splits it, and each chunk is hashed. Chunks whose hash matches what is already stored keep their vector. The rest go to embedding workers in batches sized for the GPU. Index writers upsert the new chunks and tombstone the old ones for that document in one step per shard. New chunks land in a small in-memory delta index that is searchable immediately and is merged into the main index in the background.

Read path. The API embeds the query (with the same model and the same normalisation as the chunks), resolves the user's groups, and fans out to all shards with the filter. Each shard runs vector search and BM25 search under the filter and returns its top candidates. The coordinator fuses the lists with reciprocal rank fusion, passes the top 30 to the re-ranker, and returns the best 10 with snippets. If the re-ranker is slow or down, the fused order is returned, flagged as rerank: skipped.

Deep dives

Chunking and the embedding pipeline

Options. Fixed-size windows by tokens with overlap are simple and predictable. Structure-aware splitting follows headings, paragraphs, code blocks and table boundaries and keeps units whole. Semantic splitting cuts where embedding similarity between adjacent sentences drops. It costs extra embedding calls at ingest and its gain is workload-dependent.

Choice. Split on structure first, then pack to a target of about 300 tokens, with 10 to 15% overlap only when a split falls mid-paragraph. Prepend a short breadcrumb (document title and heading path) to the text that is embedded, though not necessarily to the text that is displayed. A chunk that says "the limit is 50" is unfindable alone. Anthropic's write-up on contextual retrieval reports that adding a model-generated situating sentence to each chunk cut top-20 retrieval failures by 35% with embeddings alone and 49% when combined with BM25 (Anthropic). A cheap breadcrumb gets part of that gain without an LLM call per chunk.

Pipeline details. Hash chunk text so an edit re-embeds one chunk. Make stages idempotent by chunk_id and content_hash, and record model_id on every chunk. Pick the model with your own labelled queries, not only a leaderboard such as MTEB.

ANN index: HNSW or IVF

HNSW is a layered proximity graph. A search starts at the top sparse layer, greedily moves toward the query, and drops a layer at a time, ending with a beam search of width efSearch on the bottom layer. M sets links per node and efConstruction sets build-time effort (Malkov and Yashunin). pgvector's defaults are m = 16, ef_construction = 64 and a query-time hnsw.ef_search of 40 (pgvector). HNSW has excellent recall per millisecond but it wants the graph and vectors in RAM, and inserts are costly.

IVF clusters vectors into nlist cells by k-means. A query finds the nprobe nearest centroids and scans only those cells. With product quantization (IVF-PQ) each vector shrinks to 16 to 64 bytes, at a recall cost (Faiss). pgvector suggests starting with rows / 1000 lists up to 1M rows and sqrt(rows) above that, with probes near sqrt(lists).

Choice. For a mostly static 31M-chunk shard with RAM available, HNSW over int8 vectors with float32 re-scoring from NVMe. For heavy churn or strong filters, IVF: cell assignment is a cheap filter and an insert touches one list. When RAM cost dominates, disk graphs such as DiskANN fit. Decide by measured recall at fixed latency on your data.

Hybrid retrieval, filters and re-ranking

Why hybrid. Dense embeddings blur rare tokens such as "E1234"; BM25 handles exact terms but not paraphrase. They fail on different queries.

Fusion. Raw scores from BM25 and cosine similarity are on different scales and distributions. Reciprocal rank fusion sidesteps that: each list contributes 1 / (k + rank) per document and the sums decide the order, with k = 60 from the original paper (Cormack et al.).

Filters. Approximate indexes often apply a filter after scanning. pgvector's documentation spells out the failure: with a condition matching 10% of rows and the default ef_search of 40, only about 4 rows survive on average, and it adds iterative scans to keep going (pgvector). Options: post-filter and over-fetch (simple, unpredictable), pre-filter to an exact scan when the filtered set is small (Elasticsearch documents exact kNN for pre-filtered subsets, Elastic), or an index that is filter-aware, as in Qdrant, where payload indexes are created for the filtered fields (Qdrant). The planner should pick by estimated selectivity: very selective filters go to brute force over the matching ids, loose filters go to the graph, and in between uses iterative or filter-aware traversal. ACL filters are the dangerous case, because they are applied for every query and often very selective.

Re-ranking. A bi-encoder embeds query and passage independently, so it is fast and approximate. A cross-encoder reads them together, which is slower and much more accurate for ordering (Nogueira and Cho). Retrieve about 100 candidates cheaply, fuse, re-rank the top 30, and return 10. Anthropic's experiments added a further drop, from 49% to 67% fewer top-20 failures, from re-ranking on top of contextual embeddings and BM25 (Anthropic). Cap the rerank step with a timeout and a fallback.

Down the stack

Layer 1The contract

  • Recall contract. For the vector stage, recall@10 of at least 0.95 against exact search on a held-out query set, checked in CI and in production through a shadow exact search on 0.1% of traffic. The end-to-end contract is relevance, measured with labelled queries (nDCG@10), which approximate recall does not capture alone.
  • Consistency of versions. A query is answered entirely from one index_version. Mixed-model vectors are never compared.

Layer 2The mechanism

  • HNSW search: a node holds a vector and, per layer, a list of neighbour ids. Entry point is the top-layer node. At each layer, greedy descent: move to the neighbour closest to the query until none is closer. At layer 0, keep a candidate min-heap and a result max-heap of size efSearch, expand the closest unvisited candidate, stop when the closest candidate is farther than the worst result. A visited bitmap or hash set prevents revisits.
  • Quantization: scalar quantization maps each float to an int8 using per-dimension or global min and max. Product quantization splits the vector into sub-vectors and replaces each with the id of its nearest centroid from a small codebook, with distances computed from lookup tables. Both lose accuracy, so keep originals for re-scoring.
  • Delta layer: a small brute-force index holds recent chunks and tombstones. Queries search both and drop main hits with a newer tombstone.

Layer 3The machine

  • Memory-bound traversal. An HNSW hop reads a neighbour list and then each neighbour's vector from a random address. A DRAM miss costs about 80 to 100 ns. A float32 768-d vector spans 48 cache lines of 64 bytes, an int8 one spans 12, which is the real reason quantization speeds queries up, not only that it saves memory. A search with efSearch = 100 computes on the order of a thousand to several thousand distances, and with SIMD distance code (AVX2 or AVX-512) each is tens of nanoseconds once in cache.
  • Page cache and mmap. If vectors are memory-mapped from NVMe, a cold page costs a 4 KiB read at about 80 to 100 µs. A query touching 1,000 cold vectors would stall for roughly 100 ms. That is why the hot set must fit in RAM, and why disk-based designs like DiskANN lay out each node's vector and neighbours together so one read fetches both.
  • Fan-out tail. A query waits for the slowest of 16 shards. If one shard has a p99 of 80 ms, the chance that at least one of 16 shards is slow on a given query is about 1 − 0.99^16 ≈ 15%. Hedged requests to a replica after the p95 cut that tail.

Why it's built this way

  1. Embeddings turn meaning into geometry, but exact nearest neighbour in 768 dimensions is a scan of every vector, 3 KB each. At 500M chunks that is far too slow, so we accept approximate answers and measure how approximate.
  2. Graph search is fast only when it follows short random reads, and DRAM is about 1000× faster than NVMe for them, so the design shrinks vectors (quantization) until the working set fits in memory.
  3. Dense vectors lose exact tokens and keyword search loses paraphrase, so we run both and fuse by rank, which avoids comparing incompatible scores.
  4. Cheap retrieval casts a wide net and expensive models read few candidates, because the cost of a cross-encoder is paid per pair per query.
  5. Graphs are expensive to mutate, so freshness comes from a small mutable layer beside a large immutable one, merged in the background, the same idea as a log-structured merge tree.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
HNSW (RAM)Best recall per ms, simple tuning (efSearch)Memory heavy, costly inserts, weak with selective filtersMostly static corpus, RAM available, loose filters
IVF-PQSmall memory, cheap inserts, cell-level filteringLower recall, needs training and re-trainingBillions of vectors, high churn, strict cost limits
Hybrid with RRFRobust, no score tuningTwo indexes to run and keep in syncMixed query types (default choice)
Cross-encoder re-rankLarge relevance gain on the top resultsGPU cost per query, added latencyThe top 10 positions matter, budget allows
Vectors in the main databaseOne system, transactional with data and ACLsLower ceiling on scale and recall tuningUp to tens of millions of vectors, filters dominate

Leadership lens

Check yourself

  1. A tenant filter matches 2% of chunks. Search quality suddenly drops after you move from IVF to HNSW with post-filtering. Why, and what do you change?

    Show answer

    With post-filtering, the graph returns its top efSearch candidates (say 100) and only about 2 of them pass the filter, so results are few and poor. Use a filter-aware index or pre-filter to an exact scan when the matching set is small, or raise the scan limit with iterative scanning.

  2. Why can the vector scores and BM25 scores not simply be added, and what does RRF do instead?

    Show answer

    They have different scales and distributions, so one would dominate and the weights drift as data changes. RRF uses only the rank in each list, adding 1/(k+rank), so documents high in both lists rise without any score calibration.

  3. Compute the vector memory for 200M chunks of 1,024 dimensions as float32 and as int8, ignoring the graph.

    Show answer

    Float32 is 1,024 × 4 = 4,096 B, so 200M × 4,096 B ≈ 820 GB. Int8 is 1,024 B, so about 205 GB, a 4x saving before graph overhead.

  4. You switch embedding models and quality improves in testing, yet some production queries return nonsense. What likely happened?

    Show answer

    Query vectors from the new model were compared against chunk vectors from the old one, or the index mixes both. Index versions must be built and switched as a unit, with the model id recorded on every chunk and checked at query time.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →