Files spread across machines: distributed file systems

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A distributed file system (DFS) presents one namespace of files and directories while spreading the bytes over hundreds or thousands of machines.
The classic design (GFS, HDFS) splits files into large chunks (64 to 128 MB), replicates each chunk on three servers across racks, and keeps all metadata in a central master.
Clients ask the master where data lives, then read and write directly to chunk servers, so the master stays off the data path.
The central master is simple and fast but becomes the limit on file count and availability; later systems (Colossus, Ceph, HDFS federation) shard or distribute metadata.
DFS designs favour large sequential reads and appends over small random writes, and offer weaker semantics than a local POSIX file system.

How it works

Roles

The metadata server (GFS master, HDFS NameNode) holds the namespace tree, the mapping from each file to its ordered list of chunk ids, and the current location of every chunk replica. It keeps all of this in RAM for speed, and persists namespace changes to an operation log that is replicated before the change is acknowledged. Chunk locations are not persisted; the master rebuilds them from chunk server reports at startup (GFS paper).

Chunk servers (DataNodes) store chunks as ordinary local files, verify checksums on read, and send heartbeats with their chunk lists.

Clients cache metadata, compute which chunk contains a byte offset (offset / chunk_size), ask the master once for that chunk's replicas, and then talk only to chunk servers.

Write path

The master grants a lease on the chunk to one replica, the primary. The client pushes data along a pipeline of replicas, each forwarding to the next, so every link carries the data once. The client then asks the primary to commit; the primary picks a serial order and the secondaries apply in that order. If any replica fails, the client retries, which can leave duplicate records. GFS called this "record append, at least once"; applications deduplicated with unique ids.

Placement and re-replication

Replica placement balances durability against write cost. HDFS puts the first replica on the writer's node, the second on a different rack, and the third on another node in that second rack (HDFS architecture). That survives a full rack loss while only one copy crosses the rack boundary. When a server dies, its heartbeat stops, and the master schedules copies of every chunk that dropped below the target replica count, prioritising chunks with only one copy left. A 12 TB server failure means 12 TB of re-replication spread over the cluster.

Limits of the single master

Every file, directory and block costs about 150 bytes of NameNode heap as a rule of thumb, so 10 million single-block files need about 3 GB (Cloudera). 500 million files of 1 to 2 chunks each needs on the order of 100 to 200 GB of heap, and every open, list, or create goes through one process. This is the small files problem. Fixes include packing small files into containers (HAR, SequenceFiles, Parquet), federating the namespace across several masters, or a distributed metadata layer. Google's GFS successor, Colossus, keeps file system metadata in Bigtable behind a horizontally scalable set of curators, which let it grow more than 100 times past the largest GFS clusters (Google Cloud). The single-master ceiling is real but high: LinkedIn runs a 380 GB NameNode heap holding 1.1 billion namespace objects (LinkedIn).

High availability adds a standby master that tails the edit log from a quorum of JournalNodes and takes over via ZooKeeper, with fencing so the old master cannot keep writing (HDFS HA). Current Hadoop 3 releases also let standbys serve reads as Observer NameNodes: each client carries the last transaction id it saw, and an Observer waits until it has caught up to that id before answering, so a client never reads older state than it wrote (Observer NameNode); at LinkedIn that roughly tripled namespace throughput.

Down the stack

Layer 1The contract

HDFS promises a single-writer, append-only file: one client holds the write lease on a path, others can read concurrently, and there is no random overwrite. Visibility and durability are separate calls. hflush() guarantees that data written so far has reached every DataNode in the pipeline and is visible to new readers, but it may still sit in those machines' memory. hsync() additionally asks each DataNode to fsync. close() completes the last block and makes the file length final at the NameNode. Namespace operations (create, rename, delete) are atomic and durable once acknowledged, because the NameNode logs them to a quorum of JournalNodes first. GFS's contract was weaker: record appends were atomic but at least once, and a region written concurrently could end up consistent yet undefined, holding mingled fragments of several writes (GFS paper).

Layer 2The mechanism

HDFS splits files into blocks of 128 MB by default. On a DataNode each block is two local files: the data, and a .meta file holding a CRC32C for every 512 bytes (dfs.bytes-per-checksum). A writer sends data as packets of about 64 KB, each containing chunks with their checksums. The packet flows down the pipeline of DataNodes, and acknowledgements flow back up; the client keeps unacknowledged packets in an ack queue and, if a DataNode fails, rebuilds the pipeline from the survivors and resends. Readers verify checksums and, on a mismatch, report the bad replica to the NameNode and read another.

The NameNode persists the namespace as an fsimage checkpoint plus an edit log; with high availability, edits go to a quorum of JournalNodes and the standby replays them continuously. Block locations are never persisted. DataNodes send heartbeats every 3 seconds and periodic full block reports, and a DataNode that has been silent for about 10 minutes (twice the 5-minute recheck interval plus ten heartbeats) is declared dead and its blocks re-replicated (hdfs-default.xml). The long timeout is deliberate: it stops a brief network blip from triggering a re-replication storm. GFS used 64 MB chunks, 64 KB checksum blocks with 32-bit checksums, chunk leases of 60 seconds, and under 64 bytes of master metadata per chunk.

Layer 3The machine

Block size comes from disk physics. A hard drive seek costs around 10 ms while sequential transfer runs at 100 to 250 MB/s, so reading a 128 MB block spends about 1% of its time seeking. Small blocks would turn analytics scans into seek-bound workloads and inflate NameNode memory. Checksums every 512 bytes are cheap because modern CPUs compute CRC32C with a dedicated instruction.

The pipeline is a network design. Each byte leaves the client's NIC once and crosses each link once, instead of the client sending three copies over its own uplink. DataNodes serve reads with transferTo, which maps to sendfile on Linux and moves bytes from page cache to socket without copying through user space. When the reader runs on the same machine, short-circuit local reads pass an open file descriptor over a Unix domain socket so the client reads the block file directly. The NameNode is a JVM holding the namespace on heap, so very large namespaces mean heaps of tens to hundreds of gigabytes and garbage collection pauses that stall every client.

Why it's built this way

Commodity disks and servers fail constantly, so data is replicated across machines and racks and verified by checksum on every read.
Disks are fast sequentially and slow to seek, so files are cut into huge blocks and workloads are streaming.
Huge blocks keep the metadata small enough for one machine's RAM, which makes a single master simple and fast.
A single master must not touch data, so clients get locations once and move bytes directly with DataNodes.
Cross-rack bandwidth is scarce, so pipelined writes and rack-aware placement send each byte across the core network as few times as possible.

Diagram

sequenceDiagram
    participant C as Client
    participant M as Master
    participant P as "Primary replica"
    participant S1 as "Secondary 1"
    participant S2 as "Secondary 2"
    C->>M: which servers hold chunk 7 of /logs/a?
    M-->>C: P (lease holder), S1, S2
    C->>P: push data
    P->>S1: forward data
    S1->>S2: forward data
    C->>P: commit write
    P->>S1: apply in serial order
    P->>S2: apply in serial order
    S1-->>P: ack
    S2-->>P: ack
    P-->>C: success

Worked example

An analytics cluster has 400 data nodes, each with 12 × 16 TB disks: 77 PB raw. With 3x replication, usable capacity is about 25 PB, and at 75% target fill, about 19 PB of data.

With 128 MB blocks, 19 PB is about 150 million blocks. At roughly 150 bytes per block object plus file and directory objects, the NameNode needs around 60 to 80 GB of heap, which is workable. If the same data were stored as 1 MB files, the count would be 19 billion objects and no single NameNode could hold it. The team enforces compaction jobs that rewrite small outputs into 512 MB Parquet files.

Recovery: one node holds 192 TB × 75% = 144 TB. When it dies, the other 399 nodes each copy about 360 GB. If re-replication is throttled to 50 MB/s per node to protect production jobs, recovery takes about 2 hours, during which some blocks have only two copies. The probability of losing a second and third replica holder of the same block in that window is low, but it is why HDFS prioritises single-replica blocks first and why larger clusters move cold data to erasure coding. HDFS has supported that natively since Hadoop 3 with policies such as RS-6-3, which stores 6 blocks of data in 9 instead of 18, but erasure-coded files give up part of the contract: append() and truncate() fail, and hflush() and hsync() do nothing, so write-ahead logs and other files that need them must stay replicated (HDFS erasure coding).

In the wild

Facebook, 2012: measuring the cost of a single NameNode. Facebook's warehouse ran the largest HDFS cluster it knew of, over 100 PB of raw disk in one namespace. Reviewing incidents, it found HDFS behind 41% of warehouse incidents, but only 10% of warehouse issues and unplanned downtime would have been avoided by a highly available NameNode, while removing the NameNode as a single point of failure would cut planned downtime by about half, since every upgrade or restart stopped the cluster. Its answer, AvatarNode, wrapped the NameNode in a ZooKeeper-coordinated hot standby with manual failover, and DataNodes sent block reports to both (Facebook Engineering).
LinkedIn, 2021: scaling one namespace to a billion objects. LinkedIn's largest cluster stores 500 PB on 10,000 nodes behind one NameNode with a 380 GB heap. On the way, a DistCp job copying 100 million log files made the NameNode unresponsive to every other client, oversized block reports stalled it until each drive was split into ten virtual volumes, and a full garbage collection at that heap size would mean a multi-minute outage. Fixes included Observer reads, which tripled namespace throughput, and a separate "satellite" cluster for logs that were half the namespace but 0.07% of the bytes (LinkedIn Engineering).

Trade-offs

Option	Strengths	Weaknesses	Choose when
HDFS-style central metadata	Simple, strong namespace consistency, data locality for compute	Master memory and throughput limits, small files problem	Batch analytics on large files in your own data centre
Federated or distributed metadata (Colossus, HDFS federation)	Scales to billions of files	More complex, cross-namespace operations harder	Very large multi-tenant storage
Ceph (CRUSH placement, no central lookup)	No metadata bottleneck for objects and blocks, self-healing	Operational complexity, tuning	Unified block, object, and file storage on-premises
Cloud object storage instead	No servers to run, near-infinite capacity	No rename atomicity, no appends, higher latency	Most new cloud analytics
NFS or a single NAS	POSIX semantics, simple	Single-box limits	Small teams, home directories

Pitfalls in production

Small files. Millions of tiny files exhaust master memory and slow every listing. Compact them and set file count quotas per directory.
Master as a single point of failure. Without HA, a NameNode crash stops the whole cluster. Run active-standby with fencing and test failover.
Unbalanced disks after expansion. New nodes sit empty while old ones fill. Run the balancer continuously at a throttled rate.
Re-replication storms. A rack power loss triggers petabytes of copying. Delay re-replication for short outages and throttle network use.
Assuming POSIX semantics. Appends, concurrent writers, and rename behaviour differ from local disks. Use a single writer per file and write-then-rename patterns.

Leadership lens

Ask about file count and average file size before capacity; metadata is usually the first limit.
Decide whether you need a DFS at all; for most cloud workloads, object storage plus a table format replaces it.
Make failure-domain mapping (rack, power, switch) part of the cluster's configuration review, since placement is only as good as the topology it is told.
Treat re-replication time as a durability metric and report it.

Check yourself

Why does the client send data through a pipeline of replicas rather than to each replica directly?
Show answer
A chain uses each server's network link once, so the client's uplink is not divided three ways and total write throughput is higher.
Why does the master not persist chunk locations?
Show answer
Chunk servers are the source of truth for what they hold, so the master rebuilds locations from their reports and avoids keeping a second copy in sync.
Why do small files hurt a DFS even when total bytes are modest?
Show answer
Every file and chunk is an object in master memory and a separate metadata operation, so file count, not bytes, sets the limit.
What does HDFS rack-aware placement protect against, and at what cost?
Show answer
Losing an entire rack, at the cost of sending one replica's worth of data across the rack boundary on each write.

Sources and further study

Primary

HDFS: Observer NameNode: consistent reads from standbys, with state ids and msync().
HDFS: high availability with the Quorum Journal Manager: JournalNodes, fencing and failover.

Systems docs

HDFS erasure coding: striped layout, built-in policies and the operations EC files lose.
hdfs-default.xml: every default quoted in this lesson, from block size to heartbeat timing.
Google Cloud: a peek behind Colossus: curators, D servers and metadata in Bigtable.
Cloudera: the small files problem: where the 150-bytes-per-object rule comes from and how to pack files.

Books

Designing Data-Intensive Applications (1st ed.), Martin Kleppmann, Ch 10 "Batch Processing": MapReduce over distributed file systems, data locality and why HDFS looks the way it does.
Understanding Distributed Systems (2nd ed.), Roberto Vitillo, chapter "File storage": a walk through a cloud file store's metadata and data layers.

Other curricula

donnemartin/system-design-primer: lists GFS and HDFS as reference architectures to read, without explaining them.
Azure Architecture Center: choose a storage service: steers HDFS workloads to Data Lake Storage, an object store with a hierarchical namespace and HDFS compatibility, and HPC to managed Lustre.

Watch

Lecture 3: GFS (MIT 6.824): Robert Morris walks through the GFS paper and its consistency model.
Ceph Tech Talk: Intro to Ceph: RADOS, CRUSH placement, and how Ceph avoids a central lookup.

Free resources

The Google File System (SOSP 2003): the paper that set the pattern for large-scale DFS design.
HDFS Architecture: NameNode, DataNodes, replica placement, and the write pipeline.
Ceph (OSDI 2006): a distributed file system with distributed metadata and computed placement.
Ceph architecture: current documentation on RADOS, CRUSH, and the metadata servers.

Open in the interactive app: practice, quiz, mock interview →