Engineering Atlas › System Design Fundamentals

Peer-to-peer systems without a center

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Discovery: from flooding to DHTs

Early file-sharing networks flooded queries to neighbours with a hop limit. That works for popular content and fails for rare items while generating traffic proportional to network size. A DHT assigns every node and every key an ID in the same space (for example 160-bit hashes). Each key is stored on the nodes whose IDs are "closest" to it. In Kademlia, closeness is the XOR of two IDs read as an integer (Maymounkov and Mazières). Newer systems keep the scheme but widen the space: the IPFS DHT hashes peer IDs and keys with SHA-256 into a 256-bit space (IPFS docs). XOR distance is symmetric and has a useful property: for a given target, the bit position of the first difference tells you how far away a node is.

Routing tables and lookups

A Kademlia node keeps up to k contacts (commonly 20) for each distance range, called a k-bucket: one bucket for nodes that differ in the top bit, one for nodes that share the top bit but differ in the next, and so on. To find a key, the node asks the alpha (the paper suggests 3) closest contacts it knows for nodes even closer to the target, then asks those, in parallel. Each round at least halves the remaining distance, so a lookup in a network of 10 million nodes takes about 23 rounds in the worst case and far fewer in practice because buckets near the target are dense.

Churn and resilience

Peers disappear without notice. Kademlia prefers long-lived contacts: when a bucket is full and a new node appears, it pings the oldest entry and only replaces it if it fails to answer. The rule comes from Gnutella traces, where the longer a node had been up, the likelier it was to stay up another hour. Values are replicated to the k closest nodes, which republish them every hour, and the original publisher must republish every 24 hours or the value expires (Kademlia paper).

Peers behind NAT churn the routing table without being useful in it, because nobody can dial them. IPFS therefore starts every node as a DHT client that can query but not answer, and switches it to server mode only when it detects it is publicly dialable (IPFS docs).

Data transfer and NAT

BitTorrent splits a file into pieces (often 256 KB to 4 MB), each verified by a SHA-1 hash in the metadata (BEP 3). BitTorrent v2 replaces that with SHA-256 and a Merkle tree per file built over 16 KiB blocks, so identical files share a root hash across torrents; hybrid torrents carry both formats so old and new clients share one swarm (BEP 52). Peers exchange piece availability, request the rarest pieces first to spread them, and reciprocate upload to those who upload to them (tit-for-tat). Most peers sit behind NAT, so systems use hole punching via a rendezvous server, or relays when punching fails. WebRTC uses the same idea through ICE, which gathers local, STUN-discovered and TURN-relayed addresses and probes pairs of them until one works (RFC 8445).

Trust

Anyone can join, so an attacker can create thousands of identities (a Sybil attack) and surround a key. Defences include deriving IDs from verifiable data, limiting contacts per IP prefix, and verifying content by hash instead of trusting the peer that served it. Prefix limits are not enough on their own: Heilman and colleagues showed that a botnet of 4,600 bots, or an operator holding 32 distinct /24 blocks, could eclipse a public Bitcoin node with at least 85% probability (USENIX Security 2015).

Down the stack

Layer 1The contract

A Kademlia lookup promises to return the k live nodes closest to a key that it can find, in O(log N) rounds, with high probability rather than certainty: under heavy churn or attack it can miss. Stored values are best-effort and expire unless republished. Content integrity is not promised by peers at all; it comes from hashes the client already trusts. A BitTorrent download promises that every completed piece matches its SHA-1 hash from the metainfo file, or in v2 the per-file SHA-256 Merkle root.

Layer 2The mechanism

Kademlia defines four RPCs: PING, STORE, FIND_NODE, and FIND_VALUE. BitTorrent's Mainline DHT implements them as KRPC messages: small bencoded dictionaries sent over UDP with a transaction ID t, a type y (query, response, or error), and query names such as ping, find_node, get_peers, and announce_peer. A get_peers response includes a token that the requester must present in a later announce_peer, which stops nodes from announcing on behalf of others (BEP 5). Peers then connect over TCP or uTP and exchange a 68-byte handshake (protocol string, 8 reserved bytes, 20-byte info hash, 20-byte peer ID). After that, every message is a 4-byte length prefix and a 1-byte ID: choke, unchoke, interested, have, bitfield, request, piece, cancel. Requests ask for 16 KiB blocks within a piece.

Layer 3The machine

DHT traffic uses UDP because a lookup is a few hundred bytes per RPC: one datagram each way, no handshake, no connection state on either side. The NAT standard says a UDP mapping must not expire in under two minutes and recommends five (RFC 4787), but clients cannot assume their router complies, so they send keepalives well inside that window or lose reachability. Bulk transfer often uses uTP over UDP, with a target delay of 100 ms (BEP 29); the IETF wrote the same idea down as LEDBAT, an Experimental RFC that caps the target at 100 ms (RFC 6817). It measures one-way queuing delay and backs off as delay approaches the target, so a torrent yields to the household's video calls instead of filling the router buffer. Pieces arrive out of order, so clients preallocate files to avoid fragmentation and buffer writes to reduce random I/O. Hashing a 4 MB piece takes a few milliseconds of CPU.

Why it's built this way

  1. There is no central index, so every node needs a way to route toward any key using only local knowledge.
  2. Nodes leave constantly, so each distance range keeps several contacts and lookups query several nodes in parallel.
  3. Lookup messages are tiny and frequent, so they use UDP to avoid connection setup costs.
  4. Peers are untrusted, so data is verified against hashes from a trusted source.
  5. Uplinks are shared with other household traffic, so transfer protocols use delay-based congestion control that yields early.

Diagram

sequenceDiagram
    participant A as Node A (searcher)
    participant B as Contact B
    participant C as Contact C
    participant D as Node D (closer)
    participant E as Node E (holds key)
    A->>B: FIND_NODE(target)
    A->>C: FIND_NODE(target)
    B-->>A: k closest it knows, incl. D
    C-->>A: k closest it knows
    A->>D: FIND_NODE(target)
    D-->>A: k closest, incl. E
    A->>E: FIND_VALUE(target)
    E-->>A: value (verified by hash)

Worked example

A game studio ships a 60 GB patch to 2 million players in the first 24 hours. That is 2,000,000 x 60 GB = 120,000,000 GB, or 120 PB. Served purely from a CDN at $0.02 per GB, it costs about $2.4 million.

With a hybrid P2P launcher, the CDN seeds the first copies and players exchange pieces. Pieces are 4 MB, so the patch has 15,360 pieces, each with a 20-byte hash in a 300 KB metadata file signed by the studio. If players upload on average 1.5 times what they download from peers and peers serve 75% of traffic, CDN egress drops to 30 PB, saving about $1.8 million.

The costs: the launcher must handle NAT traversal (a relay fleet for the 10 to 20% of peers that cannot connect directly), must cap upload so it does not ruin players' connections, and must verify every piece by hash so a malicious peer cannot inject modified code. Players on metered connections need an opt-out.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Client-serverSimple, controllable, easy to secureCentral cost and capacity, single point of controlMost applications
Hybrid P2P (tracker plus peers)Offloads bandwidth, central control of membershipStill needs coordination servers, NAT complexityLarge file distribution, live video, games
Structured DHTO(log N) lookup, no central indexChurn maintenance, Sybil exposureDecentralised storage and naming
Unstructured gossip or floodingSimple, robust to churnHigh message cost, poor for rare itemsSmall networks, epidemic broadcast

Pitfalls in production

Leadership lens

Check yourself

  1. Why does Kademlia prefer old contacts over new ones in a full bucket?

    Show answer

    Nodes that have been online a long time are statistically likely to stay online, and keeping them also makes the routing table harder for an attacker to flood with fresh identities.

  2. How does BitTorrent stop a malicious peer from corrupting a download?

    Show answer

    Every piece has a hash in the metadata, so the client verifies each piece on arrival and discards and re-requests bad ones.

  3. Why do most P2P systems still run servers?

    Show answer

    Bootstrapping, NAT traversal, relaying, and trusted metadata need stable, reachable, authoritative endpoints.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →