Engineering Atlas › System Design Fundamentals

Balancing traffic through DNS

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The resolution path

A client asks a recursive resolver (an ISP's, a corporate one, or a public one like 8.8.8.8). It walks root, TLD and authoritative servers and caches each answer for its TTL. Your balancing logic lives only in the authoritative server. Everything downstream is a cache you do not control.

So the unit you balance is the resolver, not the user. The address the authoritative server sees belongs to the resolver, and one resolver may serve a single office or a whole continent (Google SRE book, ch. 19). The answer it caches decides where all of those users go until the TTL expires.

Policies the authoritative server can apply

Why TTL is the central knob

A low TTL (30 to 60 seconds) lets you shift traffic quickly but raises query volume and adds resolution round trips. A high TTL is cheap and fast for clients but locks in stale answers during an incident. Resolvers may clamp TTLs, clients may cache addresses longer, and long-lived connections never re-resolve, so a long tail lingers for hours. Resolvers that implement serve-stale go further on purpose: when they cannot reach your authoritative servers, they keep answering from expired cache with a 30-second TTL, for up to one to three days (RFC 8767). That protects you during a DNS provider outage, and it also means a record you deleted can outlive its TTL.

What DNS cannot see

The authoritative server does not know an endpoint's connection count, current latency, or request cost. It sees only resolver query rates. That is why DNS is the first tier, choosing a region or a set of load balancer IPs, with an L4 or L7 load balancer behind each address doing real balancing.

Down the stack

Layer 1The contract

The authoritative server promises only this: for a given question (name, type, class), return a set of records and a TTL saying how long the answer may be reused. A recursive resolver may serve that answer to anyone for up to TTL seconds, and may cache a negative answer (NXDOMAIN) for the smaller of the SOA record's MINIMUM field and its own TTL (RFC 2308). There is no promise that two consecutive queries get the same answer, that a client sticks to one address, or that a change is seen by everyone at once. Failure is explicit only in rare cases (SERVFAIL, timeouts); a stale answer looks exactly like a fresh one.

Layer 2The mechanism

A DNS message has a 12-byte header (16-bit ID, flags such as QR, AA, TC and RD, and four section counts), a question section, then answer resource records. Each record carries name, type, class, a 32-bit TTL, a length, and data. A load-balancing authoritative server, such as Route 53 or PowerDNS with Lua records, evaluates a policy per query: it reads health state fed by its probers, applies weights or a latency table keyed by the client's network, and builds the answer. The EDNS0 OPT pseudo-record advertises the UDP payload size the resolver accepts and can carry the Client Subnet option, truncated for privacy to a /24 for IPv4 or a /56 for IPv6. The response echoes a scope prefix telling the resolver how widely it may share that answer in its cache (RFC 7871). Newer SVCB and HTTPS records (RFC 9460, 2023) add a priority field whose equal-priority entries clients shuffle, plus hints such as alpn=h3, so a single lookup can steer both the endpoint and the protocol.

Layer 3The machine

A query is one UDP datagram of about 40 to 60 bytes; a response with four A records is about 120 bytes, since each A record with a compressed name pointer costs 16 bytes. One round trip to a nearby resolver is 1 to 20 ms; a full recursive walk on a cold cache is 50 to 200 ms. Answers above the advertised size set the TC bit and force a retry over TCP, adding a handshake round trip. The 2020 DNS Flag Day settled on 1232 bytes as a safe EDNS size, because IPv6's 1280-byte minimum MTU minus 48 bytes of headers avoids IP fragmentation, which many firewalls drop; the same campaign made answering over TCP port 53 mandatory for authoritative servers (DNS Flag Day 2020). The old 512-byte limit some textbooks still quote applies only to queries without EDNS. Authoritative servers answer in microseconds of CPU per query and scale across cores with SO_REUSEPORT sockets, so the expensive part is never the server; it is the round trips and caches between it and the user.

Why it's built this way

  1. Billions of clients cannot all ask one authority, so DNS is a hierarchy with caching everywhere.
  2. Caching means the authority loses control of an answer once it leaves, so TTL is the only steering lever.
  3. UDP gives a single round trip with no connection state, which caps message size, so answers carry a few records, not a fleet.
  4. Resolvers hide the user's location, so ECS was added to recover it, at the cost of cache fragmentation and privacy.
  5. All of this makes DNS a slow, coarse control loop, which is why precise balancing lives in L4 and L7 load balancers.

Diagram

sequenceDiagram
    participant C as Client
    participant R as "Recursive resolver"
    participant A as "Authoritative DNS"
    participant H as "Health checker"
    participant E as "Region endpoint"
    H->>A: eu-west unhealthy, remove
    C->>R: api.example.com?
    R->>A: query (with client subnet)
    A-->>R: us-east IPs, TTL 60s
    R-->>C: cached answer
    C->>E: TCP + TLS to us-east IP
    Note over R: Same answer served to all users of this resolver for 60s

Worked example

An API serves 40,000 requests per second from three regions: us-east (50%), eu-west (30%) and ap-south (20%), using latency routing with health checks and a 60-second TTL.

Query load: 20,000 active resolvers re-querying once a minute is about 330 queries per second; a 1-hour TTL drops that to about 6.

Failover timing when eu-west fails:

  1. Health checks every 10 seconds, 3 failures to trip: about 30 seconds.
  2. Provider propagates the change to its edge: 5 to 30 seconds.
  3. Resolvers hold the old answer for up to 60 seconds.
  4. Pooled connections to eu-west persist until they error.

Realistic recovery for 90% of traffic is 2 to 3 minutes. At 12,000 RPS, a 3-minute outage fails about 2 million requests, roughly 2% of a 0.1% monthly error budget (about 104 million requests). That number decides whether you need anycast or client-side retries against a second hostname.

Capacity: if Europe fails over to us-east, it carries 32,000 RPS, 60% above normal. Size headroom for losing the largest neighbour.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Round-robin DNSFree, simpleNo health awareness, uneven due to cachingSmall fleets of identical, stable endpoints
Weighted DNSGradual shifts, canariesWeights apply per resolver, not per userRegion migrations and staged rollouts
Geo/latency DNSUsers reach nearby regionsResolver location may not match userMulti-region user-facing services
DNS failoverRemoves dead regions automaticallyMinutes to convergeCross-region disaster recovery
AnycastSub-minute shifts, no client cachingNeeds BGP and owned IP spaceEdge networks, DNS itself, DDoS absorption

Pitfalls in production

Leadership lens

Check yourself

  1. Why can weighted DNS at 90/10 produce a 70/30 split of actual traffic?

    Show answer

    Weights apply per resolver query, and resolvers differ hugely in how many users they serve. A few large resolvers that cached the minority answer can carry a disproportionate share of users for the whole TTL.

  2. You set TTL to 30 seconds, yet traffic to a drained region lingers for 20 minutes. What are likely causes?

    Show answer

    Long-lived or pooled connections that never re-resolve, client libraries that cache addresses beyond TTL, and resolvers that enforce a minimum TTL.

  3. Why is DNS usually paired with an L4 or L7 load balancer rather than listing every server?

    Show answer

    DNS has no per-request view of load or health and converges slowly, so it should point at a stable, highly available entry point that does real balancing.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →