Engineering Atlas › Problem Solving & Incidents (AI era)

Facebook BGP Outage (2021)

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Timeline (from Facebook's post and Cloudflare's external observations)

Technical root cause

Two mechanisms combined. First, a command issued during backbone maintenance disconnected the entire backbone, and the audit tool meant to catch commands like that had a bug. Second, the DNS servers' health rule (stop announcing my routes if I cannot reach the data centres) was designed for the case where one DNS site has a local problem and traffic should go to healthy sites. When the backbone failed, every site failed the check simultaneously, so all of them withdrew. The safety mechanism removed every path to DNS at once.

Contributing factors

Detection and mitigation

Detection was instant from the outside. Mitigation required restoring the backbone, which required physical presence. Once the network was back, Facebook did not switch everything on at once: data centres had seen a large drop in power consumption, and a sudden return of all traffic risked electrical and cache problems (Meta Engineering). Engineers used experience from "storm" drills, which simulate large failures, to bring load back in stages.

What changed afterwards

Facebook's post describes an ongoing review to make systems more resilient, strengthening testing and drills, and extending storm drills to simulate a global backbone outage. It does not list specific code changes, so this lesson does not invent them.

What I would have done as the tech lead

Treat any command that can affect the whole backbone as gatekeeper work: a staged execution (one region first), and an audit tool that is itself tested with known-bad commands. Design health-driven withdrawal to fail static: if more than a fraction of sites fail the same check at once, assume the check (or a shared dependency) is the problem and keep announcing. Keep a genuinely independent out-of-band network and access path, and drill using it. Plan the recovery ramp in advance, since restoring huge load all at once can cause a second outage.

Down the stack

Layer 1The contract

Facebook's authoritative DNS promised that its name servers would always be reachable from somewhere on the internet, through anycast: the same IP prefixes announced from many locations, with traffic going to the nearest healthy one. Each site's withdrawal rule served that contract locally, but no rule protected it globally.

Layer 2The mechanism

BGP is how networks tell each other which IP prefixes they can reach. An anycast DNS site announces its service prefixes from its routers; if the site withdraws its announcement, neighbours stop sending traffic there and pick another site still announcing. A health check that triggers withdrawal is common: a site that cannot serve correct answers should step aside, and RFC 4786 ties route advertisement to service health, while warning that pessimistic withdrawal can take a whole node offline. The failure mode is correlated health: if the check depends on something every site shares, such as reachability of a central backbone, then one shared failure makes every site withdraw, and the prefix disappears from the global routing table.

Layer 3The machine

When a prefix is withdrawn, BGP routers across the internet remove it from their routing tables within seconds to minutes, as withdrawal messages propagate peer to peer. DNS resolvers asking for facebook.com then cannot reach any name server: their packets have no route. Resolvers return SERVFAIL and may cache it briefly; clients retry, multiplying query load on resolvers worldwide. Bringing the routes back is just as fast technically once someone can configure the routers, which is why the hours of the outage were spent on access, not on BGP.

Why it's built this way

  1. Anycast plus health-based withdrawal routes traffic away from a broken site, which is good locally.
  2. If every site's health depends on one shared component, the rule fails globally.
  3. Systems therefore need a global guard: if too many sites fail at once, keep serving.
  4. Recovery tools must not depend on the network they recover.
  5. Large systems must also be restarted gradually, because the restart itself can overload them.

Diagram

flowchart TD
  C["Maintenance command: assess capacity"] --> A{"Audit tool"}
  A -->|"bug: allowed"| BB["All backbone links down"]
  BB --> DC["Data centres disconnected"]
  DC --> H["Every DNS site fails health check"]
  H --> W["All sites withdraw BGP routes"]
  W --> X["DNS unreachable: facebook.com does not resolve"]
  BB --> OOB["Remote and out-of-band access lost"]
  OOB --> PH["Engineers go on site, secure hardware"]
  PH --> R["Backbone restored, load ramped gradually"]

Worked example

A company runs its own anycast DNS from 12 sites. Each site withdraws its route if its health check, a query to the central configuration API, fails three times in a row. A config API outage would make all 12 sites withdraw: the same shape as Facebook's.

The team redesigns the rule. Sites withdraw only for local reasons (the DNS process is down, local answers fail validation). For shared dependencies, each site keeps serving its last known good data (fail static). A coordinator also enforces a global limit: at most 3 of 12 sites may be withdrawn at a time, and at least 9 must always announce; if more sites fail at once, the extra ones keep announcing and an alert pages the on-call, because simultaneous failure usually means a shared cause. They test it in a game day by blocking the config API: all 12 sites keep answering from cached data, and one alert fires. They also verify that the out-of-band network for router access uses a separate provider and authentication path, and run a drill using only that path.

In the wild

Trade-offs

DesignStrengthsWeaknessesChoose when
Withdraw on any failed health checkRemoves broken sites quicklyCorrelated failure withdraws everythingChecks that are truly local
Fail static on shared dependency failureKeeps serving through central outagesMay serve stale dataShared config, control planes
Global limit on simultaneous withdrawalsPrevents total withdrawalCan keep a truly bad site in serviceAnycast and load balancer pools
Independent out-of-band networkAccess when production is downCost, must be maintained and testedAny critical infrastructure
Staged command executionLimits blast radius of a bad commandSlower operationsBackbone and fleet-wide changes

Pitfalls in production

Leadership lens

Check yourself

  1. Why did withdrawing DNS routes make Facebook unreachable even though its servers were running?

    Show answer

    Without route announcements for the DNS server prefixes, nobody on the internet could reach the name servers, so domain names could not be resolved to any address.

  2. Why is a health check that depends on a shared backbone dangerous for anycast withdrawal?

    Show answer

    A single backbone failure makes every site fail the check at once, so all sites withdraw together and the service disappears globally.

  3. What does "fail static" mean here?

    Show answer

    When a shared dependency fails, sites keep serving with their last known good state instead of withdrawing, because global simultaneous failure usually indicates a shared cause, not bad sites.

  4. Why did Facebook bring services back gradually?

    Show answer

    Restoring all traffic at once could cause power surges and overload caches and systems, potentially triggering a second failure.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →