Engineering Atlas › System Design Fundamentals

Anycast: one address, many locations

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

One prefix, many origins

Each point of presence (PoP) runs routers that announce, say, 198.51.100.0/24 to their transit providers and peers. Every network on the internet now holds several candidate paths to that prefix and picks one using BGP's decision process: highest local preference (usually "customers before peers before transit"), then shortest AS path, then lowest origin type, then lower MED (compared only between routes from the same neighbouring AS), then eBGP over iBGP, nearest next hop, and finally lowest BGP identifier (RFC 4271, section 9.1.2.2). Each network forwards toward its chosen path, so the set of clients that reach a given PoP, its catchment, emerges from thousands of independent policy decisions.

Failover by withdrawal

When a PoP is unhealthy, its routers withdraw the announcement. Neighbouring networks drop that path and fall back to their next-best route, typically within seconds to tens of seconds as the withdrawal propagates. No DNS answer has to expire. Health checking is local: a daemon on each edge router or server stops announcing when the service fails its checks.

Why TCP mostly works

A TCP connection needs every packet to reach the same server. Anycast cannot guarantee that, but internet routes are stable over minutes to hours for most paths, so short connections almost always complete. Measurement backs this up: probing 11 anycast services from over 9,000 vantage points, Wei and Heidemann found about 1% of vantage points flipping between sites often, and TCP flips in only about 0.15% of cases (Wei and Heidemann, 2018). Problems appear when routes flap (a peering session resets, a provider does traffic engineering) mid-connection: packets land at a PoP with no state for that connection and get a reset. Operators reduce this with stable announcements and, inside each PoP, L4 balancers that combine consistent hashing with connection tracking so a server change does not move existing flows (Maglev, NSDI 2016).

Controlling catchments

Because BGP ignores latency, operators shape catchments with tools: AS-path prepending (making a PoP look farther), selective announcement to specific peers, BGP communities that ask providers to lower preference, and announcing more specific prefixes from certain sites. Measuring catchments continuously, with probes from many networks, is part of running an anycast network. The cost of not doing so is measurable: a year of data from the D-Root server (over 400 million queries a day, 100+ sites) showed unbalanced site loads and most queries travelling farther than necessary, many by more than 5,000 km, partly because BGP tells a network nothing about how good each reachable site is (Li et al., SIGCOMM 2018).

Down the stack

Layer 1The contract

IP routing promises best-effort delivery of each packet toward some origin of the destination prefix. With anycast there is no promise which origin, no promise that consecutive packets reach the same one, and no notification when the choice changes. After a site withdraws its announcement, the internet converges on other sites in seconds to a few minutes. Anything stronger, such as connection affinity or latency targets, is an operational outcome you engineer and measure, not a guarantee.

Layer 2The mechanism

BGP speakers exchange UPDATE messages. Each carries withdrawn routes, path attributes (ORIGIN, AS_PATH, NEXT_HOP, LOCAL_PREF, MULTI_EXIT_DISC, COMMUNITIES) and the prefixes (NLRI) they apply to. A router stores what each neighbour sent in an Adj-RIB-In, runs the decision process to choose one best path per prefix into the Loc-RIB, installs it in the forwarding table (FIB), and advertises its choice onward. At each PoP, a health daemon such as ExaBGP or BIRD peers with the edge router and announces the service prefix only while local checks pass. Providers filter IPv4 prefixes longer than /24 (and IPv6 longer than /48), so anycast needs at least a whole /24. Its owner signs a Route Origin Authorization (ROA) in RPKI naming the origin AS and a maximum length, so networks that validate can drop a hijacker's more-specific route; networks that do not validate still accept it (Cloudflare, 2024). Inside the PoP, routers spread flows across servers with ECMP, hashing the 5-tuple, and L4 balancers such as Katran or Maglev use consistent hashing so a server change remaps few flows.

Layer 3The machine

BGP sessions run over TCP port 179. The standard suggests a 90-second hold time with keepalives at a third of that, a 30-second minimum advertisement interval toward eBGP neighbours and 5 seconds toward iBGP (RFC 4271, section 10). Tens of seconds is far too slow to detect a dead peer, so operators add BFD, whose detection time is a few missed packets at intervals of milliseconds (RFC 5880). Route flap damping slows propagation deliberately; RFC 7196 calls the common default suppress threshold of 2,000 too aggressive and recommends at least 6,000 (RFC 7196), so a prefix that flaps a few times during a health-check wobble can be suppressed for far longer than the wobble. Forwarding itself is hardware: a longest-prefix match in TCAM or SRAM takes nanoseconds, against a global IPv4 table that passed a million routes and stood at about 1.085 million active entries in October 2026 (BGP table report). Switch ASICs and server NICs hash packet headers to choose ECMP paths and receive queues (RSS), which is why flows stay pinned as long as nothing in the path changes.

Why it's built this way

  1. Services need one stable address reachable from everywhere, and IP already chooses among paths, so reuse routing instead of inventing a directory.
  2. BGP chooses paths by business policy (customers before peers before paid transit), so "nearest" follows money and AS hops, not milliseconds.
  3. Routers carry no per-flow state, so nothing keeps a TCP connection on one site; anycast suits short or stateless exchanges.
  4. Router memory is finite and the global table is huge, so the minimum routable prefix is /24, which makes anycast a commitment of real address space.
  5. Fast route changes destabilise the whole internet, so timers and damping trade failover speed for global stability.

Diagram

flowchart LR
    U1["Client in Paris"] --> ISP1["ISP A"]
    U2["Client in Tokyo"] --> ISP2["ISP B"]
    ISP1 -->|"AS path 2"| P1["PoP Frankfurt 198.51.100.0/24"]
    ISP1 -.->|"AS path 4"| P2["PoP Singapore 198.51.100.0/24"]
    ISP2 -->|"AS path 2"| P2
    ISP2 -.->|"AS path 3"| P1
    P1 -->|"health fails: withdraw"| ISP1

Worked example

A DNS provider runs 30 PoPs, each with 100 Gbps of transit, serving 2 million queries per second at about 120 bytes per response.

Normal traffic is about 2M x 120 B x 8, roughly 2 Gbps in total, so capacity is mostly there for attacks. A 1.2 Tbps reflection attack sourced from botnets in many countries spreads across catchments: if it splits roughly in proportion to where bots sit, the busiest PoP may take 15% (180 Gbps) and saturate, while the rest absorb 30 to 60 Gbps each. The operator withdraws the prefix from the overloaded PoP's most congested transit, shifting part of its catchment to neighbours, or announces only to peers there.

Failover timing: health daemon detects the failure in 3 seconds, BGP withdrawal reaches most of the internet in 10 to 60 seconds. Compare that with DNS failover, where a 60-second TTL plus resolver behaviour gives several minutes.

Suboptimal routing: probes show that 8% of clients in Brazil reach the Miami PoP (130 ms) instead of São Paulo (15 ms) because their provider prefers a transit path. The fix is peering directly with that provider in São Paulo, not tuning servers.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Global anycastFast failover, DDoS dilution, one IPBGP ignores latency; needs owned prefix and ASNDNS, CDN edges, public resolvers
DNS geo routingFine-grained control per regionSlow failover, resolver location errorsOrigin region selection behind the edge
Regional anycastSmaller blast radius, easier capacityMore IPs to manageIsolating regions for compliance or capacity
Anycast to edge, unicast to originStateless edge, stable backendTwo tiers to operateMost CDN and API edge designs

Pitfalls in production

Leadership lens

Check yourself

  1. Why can anycast send a user to a farther PoP than the nearest one?

    Show answer

    BGP picks paths by policy and AS-path length, not latency. A provider may prefer a transit path to a distant PoP over a peering path to a close one.

  2. Why is anycast failover faster than DNS failover?

    Show answer

    Withdrawal propagates through routers within seconds, and there is no client or resolver cache holding the old answer.

  3. What happens to an open TCP connection if routing shifts to another PoP?

    Show answer

    Packets arrive at a server with no connection state and it responds with a reset, unless the operator forwards strays to the original PoP.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →