Engineering Atlas › System Design Fundamentals

How IP addresses identify machines

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

CIDR: prefixes and the arithmetic

Classless Inter-Domain Routing (1993) replaced fixed class A, B and C networks with a prefix of any length. A /n block holds 2^(32−n) addresses: /16 is 65,536, /24 is 256, /28 is 16. Blocks align on their size, so 10.0.4.0/22 covers 10.0.4.0 to 10.0.7.255, while 10.0.5.0/22 is invalid because host bits are set. Adjacent aligned blocks aggregate: 10.0.4.0/23 and 10.0.6.0/23 together form 10.0.4.0/22. Aggregation is what keeps the global IPv4 table at about 1.05 million routes (January 2026), instead of one per network (APNIC, BGP in 2025).

A router picks the most specific match: with 0.0.0.0/0 → internet, 10.0.0.0/8 → transit and 10.20.3.0/24 → VPN, a packet to 10.20.3.9 takes the VPN. Traffic engineering and hijacks both exploit this: announce a more specific prefix and you win the traffic. The defence is RPKI route origin validation. The prefix holder publishes a signed ROA naming the AS allowed to originate it and the longest prefix length allowed, and routers that enforce it drop announcements that don't match (RFC 6811). Many networks also refuse IPv4 prefixes longer than /24. Both only help where operators actually enforce them.

Reserved ranges matter: private 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 (RFC 1918); carrier-grade NAT 100.64.0.0/10 (RFC 6598), which some Kubernetes setups borrow for pods; link-local 169.254.0.0/16, home of the cloud metadata endpoint 169.254.169.254.

Subnets in a cloud network

An AWS VPC takes one or more blocks between /16 and /28. Each subnet lives in one availability zone and loses five addresses: network, VPC router, DNS, one reserved for future use, and broadcast (AWS subnet sizing). Each zone typically gets a public subnet (load balancers, NAT), a large private one for compute, and a small one for databases.

Two decisions are hard to undo. First, ranges must not overlap anything you might ever connect to (other VPCs, on-premises, partners, acquisitions), because routing cannot distinguish identical prefixes. Second, size for pods, not nodes. With the AWS VPC CNI, every pod gets a real VPC address, so a 300-node cluster at 30 pods per node consumes 9,000 addresses.

NAT: sharing addresses with state

Source NAT rewrites an outgoing packet's source address and port to a public address and a free port, and records the mapping so replies can be rewritten back. The translator keys each flow on the 5-tuple, so one public IP can reuse the same port for flows to different destinations. To one destination IP and port, though, it can hold at most about 64,000 flows. AWS NAT Gateway documents 55,000 simultaneous connections per unique destination (destination IP, port and protocol) for each public IP, and allows up to eight IPv4 addresses per gateway (AWS NAT gateway basics). A flow idle for 350 seconds times out, and the gateway answers the next packet with an RST, not a FIN (AWS troubleshooting).

Unsolicited inbound packets have no mapping and are dropped, which people mistake for a firewall. Peer-to-peer apps work around it with STUN and hole punching, falling back to TURN relays when the NAT maps per destination. RFC 4787 classifies these behaviours and requires UDP mappings to last at least two minutes, although many deployed NATs time out sooner.

Unicast, anycast, multicast

Most addresses are unicast. Anycast announces one prefix from many sites so routing picks the nearest (see the anycast lesson). Multicast (224.0.0.0/4, ff00::/8) delivers to a group, common inside data centres and rare across the internet.

IPv6

IPv6 addresses are 128 bits, written in hex groups with :: for one run of zeros, for example 2001:db8:12::7. Every LAN is a /64, so hosts can pick addresses themselves (SLAAC) and rotate privacy addresses. Sites typically get a /48 or /56. ARP is replaced by Neighbor Discovery over ICMPv6, so blocking ICMPv6 breaks the network outright. The header is a fixed 40 bytes with no checksum, and only the source may fragment, never a router (RFC 8200).

Native IPv6 access to Google first passed 50% on 28 March 2026, though the average across the measurements the Internet Society tracks is nearer 43% (ISOC Pulse). Many mobile carriers run IPv6-only with NAT64, and AWS charges $0.005 an hour, about $3.65 a month, for every public IPv4 address (AWS VPC pricing). Clients that resolve both A and AAAA records use Happy Eyeballs (RFC 8305). They wait up to 50 ms for the AAAA answer, try IPv6, start IPv4 about 250 ms later if IPv6 has not connected, and use whichever finishes first, so a broken IPv6 path costs latency instead of an outage. Version 3, still an Internet-Draft in 2026, folds HTTPS/SVCB DNS records and QUIC into the same race (draft-ietf-happy-happyeyeballs-v3).

Down the stack

Layer 1The contract

Routing promises that a packet sent to an address goes, best effort, towards whichever interface currently holds the longest matching prefix. It does not promise that the holder is who you think. Source addresses are not authenticated, and spoofing is stopped only where networks filter at the edge (BCP 38). A NAT promises that an outbound flow gets a stable public mapping while it stays active, that replies come back through that mapping, and that unsolicited inbound traffic is dropped. A mapping idle for longer than the timeout is gone, and the next packet on that flow gets a reset or silence.

Layer 2The mechanism

Longest-prefix match needs a structure better than a hash table, because the key length varies. Linux uses an LC-trie (fib_trie) that compresses the binary trie into multi-way nodes. DPDK's rte_lpm uses DIR-24-8: a 2^24-entry table indexed by the first 24 bits, plus 256-entry second-level tables for longer prefixes, so most lookups take one memory read. Hardware routers use TCAM, which compares the address against every entry in parallel and returns the highest-priority match.

The next hop's MAC comes from ARP or Neighbor Discovery, cached in ip neigh. Linux NAT runs in netfilter conntrack. Each nf_conntrack entry stores the original and reply tuples, state and a timeout (by default 5 days for established TCP, 30 seconds for one-way UDP, 120 seconds for UDP streams, per the kernel docs). When the table reaches nf_conntrack_max, the kernel logs "table full, dropping packet", and new flows fail.

Layer 3The machine

The global table holds about 1.05 million IPv4 routes and about 240,000 IPv6 routes. The original DIR-24-8 design used 2-byte entries, so its first level alone is 32 MB; DPDK's entries are 4 bytes, so it is 64 MB. That explains why software routers care about huge pages and cache misses. A miss to DRAM costs about 100 ns, while the per-packet budget at 10 Gbps line rate with small packets is 67 ns. TCAM answers in nanoseconds but costs far more power and money per bit than SRAM, so table growth is watched closely.

A conntrack entry costs a few hundred bytes, so a million flows is hundreds of megabytes of state. On the wire, an IPv4 address is 4 bytes and an IPv6 address 16, so IPv6 adds 20 bytes of header per packet: 1.3% of a 1,500-byte packet, but 50% of a 40-byte TCP ACK.

Why it's built this way

  1. In 1981, 32 bits looked plentiful, and fixed classes wasted most of each block, so CIDR (1993) introduced arbitrary prefixes to slow exhaustion.
  2. Routers cannot hold a route per host, so addresses follow topology and are aggregated, which makes an address a location, not a name.
  3. Addresses still ran out, so NAT spread one address across many hosts, at the cost of per-flow state and end-to-end reachability.
  4. NAT's costs and continued growth justified IPv6. Changing every host, router and application takes decades, so dual-stack and Happy Eyeballs bridge the gap.

Diagram

sequenceDiagram
    participant P as "Pod 10.0.32.15"
    participant N as "NAT GW 203.0.113.9"
    participant A as "API 198.51.100.20:443"
    P->>N: src 10.0.32.15:43211
    Note over N: allocate port 1025, store mapping
    N->>A: src 203.0.113.9:1025
    A->>N: dst 203.0.113.9:1025
    Note over N: look up mapping, rewrite dst
    N->>P: dst 10.0.32.15:43211
    Note over N: idle 350 s: mapping removed

Worked example

A company plans a 10.40.0.0/16 VPC for an EKS platform across three availability zones, expecting 25,000 pods at peak with the VPC CNI.

Egress is next. 2,000 pods each keep up to 40 concurrent connections to a payment provider at a single address on port 443. That is 80,000 flows to one destination through one NAT IP limited to 55,000, so about a third of new connections fail with port allocation errors during peaks. CloudWatch shows ErrorPortAllocation climbing. There are three fixes, in order of cost. First, connection pooling (a shared HTTP/2 client per pod brings flows down to 2,000). Second, more public IPs on the NAT gateway (each adds 55,000, up to eight per gateway). Third, IPv6 egress, if the provider supports it.

Cost: NAT data processing is about $0.045 per GB, so 40 TB a month is about $1,800 before transfer charges. An S3 gateway endpoint often removes most of that volume.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Large private IPv4 VPC per environmentSimple routing, pods get real IPsConsumes scarce RFC 1918 space, overlap riskFew environments, central IP plan
Overlay or 100.64/10 pod rangeSaves routable spaceEncapsulation or SNAT, harder debuggingVery large clusters, constrained IP plan
NAT gateway egressNo public IPs on hosts, managedPer-destination port limits, per-GB cost, stateDefault private-subnet egress
Public IP per hostNo NAT state or limitsIPv4 cost, larger attack surfaceHigh-connection-count egress, P2P
IPv6 dual-stackEnds scarcity and NAT for v6 peersTwo of everything: rules, monitoring, testingNew platforms, mobile-heavy traffic

Pitfalls in production

Leadership lens

Check yourself

  1. Routes exist for 10.0.0.0/8, 10.1.0.0/16 and 10.1.2.0/24. Where does a packet to 10.1.3.4 go, and why?

    Show answer

    Via 10.1.0.0/16. The /24 does not contain it, and of the prefixes that match, the /16 is the longest.

  2. Why can one NAT IP hold far more than 65,535 connections in total, yet fail at 55,000 to one API?

    Show answer

    Mappings are unique per full 5-tuple, so a port can be reused for different destinations. With the destination IP and port fixed, only the source port varies, which caps flows to that destination.

  3. Why does IPv6 make NAT unnecessary but not firewalls?

    Show answer

    Scarcity was NAT's reason to exist, and every subnet now has 2^64 addresses. The "drop unsolicited inbound" behaviour people relied on must now be an explicit stateful firewall rule.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →