Engineering Atlas › Problem Solving & Incidents (AI era)

AWS us-east-1 DynamoDB DNS Outage (2025)

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Timeline (PDT, from AWS's summary)

Technical root cause

DynamoDB's DNS automation has two parts (AWS summary). The DNS Planner watches load balancer health and capacity and produces versioned DNS plans: which load balancers, with which weights, for each endpoint. DNS Enactors, running redundantly in three availability zones, apply plans to Route 53 using transactions, and a cleanup process removes plans much older than the latest applied one.

The race: one Enactor was unusually slow, retrying updates on several endpoints. Meanwhile, the Planner produced many newer plans, and a second Enactor applied the newest plan quickly, then ran cleanup. The slow Enactor, whose check that its plan was newer than the current one had been made long before, then wrote its much older plan over the regional endpoint. Cleanup deleted that old plan because it was far older than the newest applied one, but it was now the active plan. Deleting it removed all IP addresses for dynamodb.us-east-1.amazonaws.com, and the inconsistent state blocked further automatic updates.

Contributing factors

Detection and mitigation

Failures appeared immediately; locating DNS as the source took about 50 minutes (23:48 to 00:38) (AWS summary). DNS was repaired manually. Each later stage needed its own mitigation: throttling and selective restarts for DWFM, and disabling automatic failover for NLB health checks.

What changed afterwards (per AWS)

The DynamoDB DNS Planner and Enactor automation was disabled worldwide; before re-enabling it, AWS will fix the race condition and add protections against applying incorrect DNS plans. NLB gets a velocity control limiting how much capacity a single load balancer can remove when health checks trigger zone failover (AWS summary). EC2 adds a test suite for the DWFM recovery workflow and improves throttling to rate-limit incoming work based on queue size. AWS said it would keep reviewing the event across services.

What I would have done as the tech lead

Make plan application a single atomic compare-and-set on the generation number at write time, and make cleanup refuse to delete whatever is currently active, checked against live state. Add an invariant monitor that pages when any endpoint has zero records. For dependent systems, design recovery paths that are rate-limited by default and tested at full fleet scale, and cap how much capacity any automatic health mechanism may remove at once.

Down the stack

Layer 1The contract

DynamoDB's regional endpoint promises that its DNS name always resolves to healthy load balancers. The DNS automation's implicit contract is monotonic: the endpoint only ever moves to newer plans, and no step leaves it empty. The race broke both.

Layer 2The mechanism

The bug is a classic time-of-check to time-of-use (TOCTOU) race, the same shape as check-then-act in a database. Two safe designs exist. Compare-and-set: write the plan only if the stored generation is still lower than mine, in the same transaction as the write; Route 53 change batches are all-or-nothing (Route 53 API reference), but that alone does not compare a generation, and AWS's summary does not say how the Enactors guard their writes, so treat this as a design sketch and not a description of their code. Fencing tokens: every writer carries a monotonically increasing token, and the store rejects writes with an older token, which is how Kleppmann says to protect against a paused or delayed writer. Separately, garbage collection of plans must be relative to the live state: never delete the plan currently referenced by any endpoint, whatever its age.

Layer 3The machine

DNS failure spreads through caching. Resolvers and clients cache answers for the record's TTL; when the record became empty, clients that refreshed got no addresses, while some cached good answers until expiry. Recovery works the same way in reverse: after DNS was restored at 02:25, connections recovered over the following minutes as caches expired (02:40). The congestive collapse in DWFM is a queueing phenomenon: when work arrives faster than it can be completed within its timeout, timed-out work is retried, adding load, so useful throughput falls towards zero even though machines are busy. Throttling admission below capacity is the way out (AWS Builders' Library on load shedding), which is what the engineers did at 04:14 and what AWS said it would build into EC2's propagation path as queue-size-based rate limiting. This is the metastable failure pattern described by Bronson et al.: a trigger is gone, but a feedback loop keeps the system down until load is removed.

Why it's built this way

  1. DNS automation must react quickly to load balancer health, so it runs continuously and redundantly.
  2. Redundant writers need atomic, generation-checked updates, or they race.
  3. Cleanup must respect live state, or it can delete what is in use.
  4. Shared dependencies turn one service's failure into many, so dependents need graceful degradation.
  5. Recovery itself can overload systems, so it must be throttled and rehearsed.

Diagram

sequenceDiagram
  participant P as DNS Planner
  participant A as Enactor A (delayed)
  participant B as Enactor B
  participant R as Route 53 endpoint
  P->>A: plan gen 100
  A->>A: check: 100 newer than current? yes
  P->>B: plans up to gen 140
  B->>R: apply gen 140
  A->>R: apply gen 100 (stale check)
  B->>B: cleanup plans far older than 140
  B->>R: delete gen 100 (now active)
  Note over R: endpoint has no IP addresses

Worked example

A platform team runs a config distributor: three replicas pull versioned routing configs from a planner and write them into a shared key-value store that edge proxies read. Each replica does current = get(); if new.version > current.version: put(new), and a janitor deletes configs more than 50 versions behind the newest.

After reading AWS's summary, they reproduce the race in a test: replica A reads version 200, pauses (simulating a long GC pause), replica B writes 260, the janitor deletes everything below 210, then A writes 200. The proxies now point at a deleted config and fall back to an empty route table.

Fixes: put becomes a conditional write (put_if(version_greater_than_stored)), executed atomically by the store; the janitor reads which version is live and never deletes it or anything newer; an invariant check alerts within 30 seconds if any proxy reads an empty or missing config; and proxies keep serving their last good config if the new one is missing (fail static). The reproduction test now ends with version 260 live, and it runs in CI.

In the wild

Trade-offs

DesignStrengthsWeaknessesChoose when
Check then writeSimpleRaces with concurrent or delayed writersSingle writer only
Atomic compare-and-set on generationNo stale overwriteNeeds store supportMultiple writers to shared state
Fencing tokensRobust to pauses and partitionsStore must check tokensLeases, leaders, delayed workers
Cleanup by age onlySimpleCan delete live dataNever for referenced data
Cleanup relative to live referencesSafeMust read live stateGarbage collection of configs or plans

Pitfalls in production

Leadership lens

Check yourself

  1. What made the delayed Enactor's write dangerous?

    Show answer

    Its "is this newer?" check was made long before its write, so by the time it wrote, a much newer plan was active and it overwrote it with an old one.

  2. Why did cleanup make the situation worse?

    Show answer

    It deleted plans much older than the newest applied one, without checking which plan was actually active, so it deleted the active plan and left the endpoint empty.

  3. What is congestive collapse, and how was it stopped?

    Show answer

    When work arrives faster than it can finish within its timeout, timeouts and retries add load until throughput collapses; engineers throttled incoming work and restarted hosts to drain queues.

  4. Why did NLB health checks remove healthy capacity?

    Show answer

    New instances' network state had not propagated, so health checks failed for healthy nodes and automatic zone failover removed them from service.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →