Engineering Atlas › System Design Fundamentals

Warehouse meets lake: the lakehouse

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Metadata over files

A plain lake defines a table as "all files under this prefix", which breaks with concurrent writers and partial writes. A table format instead defines the table as an explicit list of data files recorded in metadata. In Iceberg, a catalog points to the current metadata file; that file lists snapshots; each snapshot points to a manifest list; manifests list data files along with per-column statistics and partition values. Delta keeps an ordered transaction log of JSON commits with periodic Parquet checkpoints. The design was argued for in the 2021 CIDR paper that named the lakehouse: one open storage layer with transactions, instead of a lake plus a copied warehouse (Armbrust et al., CIDR 2021).

Atomic commits with optimistic concurrency

A writer writes new Parquet files first (invisible, since nothing references them), then writes new metadata, and finally tries to swap the table pointer from version N to N+1 with a compare-and-swap in the catalog, or by creating log file N+1 only if it does not exist. If another writer committed first, the swap fails; the writer re-reads, checks whether the changes conflict (for example both rewrote the same files), and retries. Iceberg retries 4 times by default, with backoff from 100 ms up to a minute, before failing the job (Iceberg docs). Appends rarely conflict; concurrent updates to the same partition do.

Updates and deletes

Files are immutable, so a row-level change is either copy-on-write (rewrite every file containing an affected row; slow writes, fast reads) or merge-on-read (write small delete files or delta files that readers apply at query time; fast writes, slower reads until compaction). Copy-on-write is Iceberg's default for deletes, so merge-on-read is a per-table choice you make (Iceberg docs).

Snapshots and time travel

Because old metadata and files are kept until expired, a query can read VERSION AS OF 412 or a timestamp. This makes rollback after a bad job a metadata operation, and it makes ML training reproducible. Expiring snapshots older than a retention window then deletes unreferenced files; Iceberg's default maximum snapshot age when expiring is 5 days (Iceberg docs).

Medallion layering

A common layout is bronze (raw ingested), silver (cleaned, conformed), and gold (aggregated business tables), all as table-format tables, so streaming and batch jobs read and write the same storage.

Down the stack

Layer 1The contract

A table format promises serializable or snapshot-isolated commits per table: each commit produces a new numbered snapshot that is either fully visible or not at all, readers always see one complete snapshot, and concurrent writers that conflict are detected and one of them must retry. Old snapshots remain readable until expired. It does not promise multi-table transactions (most catalogs cannot commit two tables atomically), and it does not promise good performance without maintenance.

Layer 2The mechanism

In Iceberg, the catalog stores one pointer per table: the path of the current metadata.json. That file lists the schema history, partition specs, and snapshots; each snapshot names a manifest list (an Avro file), which lists manifests with partition value ranges, and each manifest lists data files with record counts and per-column lower and upper bounds. Planning reads the manifest list, skips manifests whose partition ranges cannot match, then skips files using column bounds, so a table of a million files can be planned by reading a few MB. A commit writes a new metadata.json and asks the catalog to swap the pointer only if it still equals the version the writer started from. The spec leaves that atomic operation to the catalog: a check-and-put in a metastore or database, or an atomic rename on file systems like HDFS (Iceberg spec). Delta Lake instead writes _delta_log/00000000000000000042.json with put-if-absent semantics; whoever creates file 42 first wins, and a periodic checkpoint summarises the log (the protocol leaves the interval to writers) (Delta protocol). S3 had no put-if-absent for years, so Delta on S3 allows writes from one Spark driver by default and needs a DynamoDB-backed log store for multi-cluster writers (Delta docs); S3 added If-None-Match conditional writes only in August 2024 (AWS). Merge-on-read deletes are separate files: position deletes list (file_path, row_position) pairs, and equality deletes list column values to drop. Format v3 replaces new position delete files with deletion vectors, a bitmap of deleted row positions with at most one per data file per snapshot, stored in Puffin files; v3 writers must not add position delete files (Iceberg spec). Delta has the same idea, also called deletion vectors. Version 4 of the Iceberg spec, still under development, would stop new equality delete files too.

Layer 3The machine

The commit is a handful of small writes, a few KB each, plus one conditional request to the catalog; at the roughly 100 to 200 ms AWS quotes for small S3 requests (S3 docs), a commit costs around a second regardless of data size. That fixed cost is why tables sustain roughly one commit per second per table rather than thousands, and why streaming writers batch for 30 to 60 seconds. On read, every equality delete file must be loaded and its keys held in a hash set in memory, then each scanned row is probed against it; a few million delete keys means hundreds of MB of RAM per task, which is the real reason merge-on-read tables need compaction.

Why it's built this way

  1. Object stores cannot rename directories atomically, so atomicity must come from one small pointer swap.
  2. Listing millions of objects is slow, so the table records its own file list with statistics.
  3. Files are immutable and large, so row changes are either rewrites or side files, never in-place edits.
  4. Keeping old metadata is cheap, so time travel comes almost for free, but storage grows until snapshots expire.

Diagram

sequenceDiagram
  participant W as Writer
  participant S as Object storage
  participant C as Catalog
  participant R as Reader
  W->>S: write new Parquet data files
  W->>S: write manifest and metadata v13
  W->>C: CAS pointer v12 to v13
  C-->>W: ok (or conflict, then retry)
  R->>C: get current pointer
  C-->>R: v13
  R->>S: read metadata, prune via stats, read files

Worked example

A payments company stores a 40 TB transactions table, partitioned by day, as Iceberg on S3. About 20 million new rows arrive per day, and compliance requires deleting a customer's records within 30 days of a request. They receive about 2,000 requests per day.

With copy-on-write, deleting one customer touches every file containing their rows. Customers transact across years, so one request might touch 600 files of 256 MB, about 150 GB of rewrite. Batched daily for 2,000 customers, the union might hit 40 percent of all files, roughly 16 TB of rewrite per day. That is far too expensive.

With merge-on-read, the daily job writes equality delete files keyed on customer_id, a few MB total. Readers filter those rows at scan time, adding a few percent of overhead. A weekly compaction rewrites only files where deleted rows exceed 10 percent, so each file is rewritten at most once a week, in a quiet window, and the rest stay filtered at read time.

The team also expires snapshots older than 7 days; otherwise "deleted" rows remain readable through time travel, which would fail the compliance requirement.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Lakehouse with IcebergEngine-neutral, hidden partitioning, strong specCatalog choice matters; maintenance jobsMulti-engine platforms avoiding lock-in
Lakehouse with DeltaMature in Spark, simple log designBest features tied to one ecosystemSpark-centric teams
Lakehouse with HudiStrong upsert and incremental pullMore configuration complexityCDC-heavy ingestion with frequent upserts
Lake plus separate warehouseBest SQL performance and governance in the warehouseTwo copies, two pipelines, driftStrict BI SLAs, little ML on raw data
Warehouse onlySimplest operationsProprietary storage, expensive for raw and ML dataSmall to medium data, SQL-only consumers

Pitfalls in production

Leadership lens

Check yourself

  1. Why can a reader never see a half-written commit in a table format?

    Show answer

    New data files are unreferenced until the metadata pointer swap, which is atomic, so readers see either the old snapshot or the new one.

  2. When does merge-on-read beat copy-on-write?

    Show answer

    When updates or deletes are frequent and small relative to file size, because writing a small delete file is far cheaper than rewriting whole data files.

  3. Why does snapshot expiry matter for privacy deletion?

    Show answer

    Older snapshots still reference files containing deleted rows, so the data stays reachable through time travel until those snapshots expire and files are removed.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →