Engineering Atlas › Problem Solving & Incidents (AI era)

Disaster Recovery: RTO, RPO and Restore Drills

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

RPO and RTO

RPO is set by how often you capture data and ship it somewhere safe. Nightly backups give an RPO of up to 24 hours; continuous log shipping gives seconds to minutes. RTO is set by how long it takes to get a working system: provision infrastructure, restore data, replay logs, verify, switch traffic. Both should come from the business: what does an hour of downtime cost, and what does losing an hour of orders cost? Different systems get different targets; the payments ledger and the internal wiki should not share a DR plan.

Strategies

AWS's Well-Architected guidance lists four strategies in increasing order of cost and complexity and decreasing RTO and RPO; they apply on any platform (AWS Well-Architected REL13-BP02):

One trap applies to all four: replication copies mistakes as faithfully as data. A replica will happily replicate a DROP TABLE or a corrupted write, so you also need versioning or point-in-time recovery, and backups of the replicated data in the recovery site (same AWS page). Google's SRE book argues for the same layering: soft deletion first, then backups, then archives, and it says to deliver a recovery system, not a backup system, because what users need is the restore (SRE book, ch. 26).

What a disaster takes with it

A DR plan fails most often on what it forgot. If your identity provider, DNS, secrets manager, container registry, CI system or monitoring live in the failed region or account, you may not be able to log in, deploy or see anything. List every dependency of the recovery procedure itself, and make sure each is available in the disaster scenario. AWS lists "dependency on control plane operations during recovery" as an anti-pattern: during a regional event the control plane is the part most likely to be struggling, so failover should use data plane operations and be scripted so that manual initiation is a push of a button. Automatic failover on health checks is risky for the opposite reason, since a false alarm costs availability and data (AWS REL13-BP02). Plan failback as well: re-synchronising the old primary is the hard part, and some teams simply promote the recovery region and rotate roles on a schedule.

The 3-2-1 rule and immutability

Keep at least three copies of important data, on two different kinds of storage, with one off-site. Today "off-site" means another region and, more importantly, another security boundary: a separate cloud account with its own credentials, and immutable storage that cannot be deleted before its retention period. With S3 Object Lock that guarantee holds only in compliance mode, where no user, including the account's root user, can overwrite or delete a locked version; the only way to remove it early is to delete the whole AWS account. Governance mode can be bypassed by principals with special permissions (S3 Object Lock docs). Ransomware and destructive mistakes, by humans or agents, target the same credentials that manage production.

Restore drills

A restore drill is a scheduled, realistic recovery: restore last night's backup into a clean environment, bring the application up, run checks, and time every step. Drills find the problems that make real recoveries fail: backups that were never completing, missing encryption keys, runbooks that reference retired tools, restores that take 14 hours instead of 2. Record measured RTO and RPO and compare them with targets.

The same shape everywhere

DR is the partial-failure shape at the largest scale: you lose a whole failure domain and must continue from what survived elsewhere. Businesses plan the same way for losing an office, a supplier or a key person: decide acceptable loss and downtime, keep copies elsewhere, and rehearse.

Down the stack

Layer 1The contract

Each system has documented RPO and RTO targets agreed with the business, a recovery runbook, backups stored in a separate security boundary, and a drill record showing measured RPO and RTO within targets at least once a year (quarterly for critical systems).

Layer 2The mechanism

Database backups combine base backups with continuous change capture. In Postgres, a base backup (for example from pg_basebackup or a managed snapshot) plus archived WAL segments allows point-in-time recovery to any moment since the base backup (Postgres docs); RPO is bounded by how often WAL is archived (archive_timeout forces a segment switch after that many seconds of activity, because the archive command only runs on completed segments (Postgres docs); for example every 60 seconds). Cross-region replicas ship WAL continuously and are usually seconds behind. Object storage can replicate across regions asynchronously. Infrastructure as code (Terraform, CloudFormation) lets you rebuild networks, clusters and services in a new region from versioned definitions. Traffic moves by DNS changes or global load balancers.

Layer 3The machine

Restore time is mostly bytes over bandwidth and log replay. Copying a 2 TB backup across regions at a sustained 500 MB/s takes about 70 minutes; at 100 MB/s, nearly 6 hours. Replaying a day of WAL depends on write volume and disk speed: a database that writes 50 GB of WAL a day may replay at 100 to 200 MB/s on fast SSDs, so about 5 to 10 minutes, but replay applies records in order and can be slower with many random page reads; recent Postgres versions can prefetch referenced blocks during recovery (recovery_prefetch, on by default where the OS supports it), which helps with that I/O wait (Postgres docs). DNS failover is bounded by record TTLs: clients and resolvers may keep the old address until the TTL expires, so a 300-second TTL means up to 5 minutes of traffic going to the dead region.

Why it's built this way

  1. Redundancy within a region does not cover regional, account-level or logical disasters.
  2. Copies elsewhere bound data loss (RPO), and rebuild speed bounds downtime (RTO).
  3. Faster recovery costs more, so targets come from business impact per system.
  4. Recovery procedures rot as systems change, so only drills prove they work.
  5. Attackers and mistakes reach whatever production credentials reach, so backups must live beyond them.

Diagram

flowchart LR
  P["Primary region"] -->|"base backup nightly"| B["Backup account, immutable"]
  P -->|"WAL every 60 s"| B
  P -->|"async replica"| W["Warm standby region"]
  D["Disaster in primary"] --> F{"Strategy"}
  F -->|"warm standby"| W
  F -->|"backup and restore"| R["Rebuild from IaC, restore, replay WAL"]
  W --> T["Shift traffic: DNS or global LB"]
  R --> T

Worked example

An e-commerce company sets targets with the business: checkout and orders RPO 5 minutes, RTO 1 hour; product catalogue RPO 24 hours, RTO 4 hours; internal analytics RPO 24 hours, RTO 3 days. Current state: one region, nightly snapshots in the same account.

For orders, they add continuous WAL archiving every 60 seconds to an immutable bucket in a separate account and region, plus a cross-region read replica that can be promoted (warm standby for the database, pilot light for the application). For the catalogue, nightly backups copied to the backup account. Analytics stays as is.

The first drill simulates losing the primary region. Results: database promotion in the recovery region took 4 minutes; but deploying the application took 48 minutes because the container images were only in the primary region's registry, and the team could not log in to the cloud console for 15 minutes because single sign-on depended on a service in the failed region. Measured RTO: 1 hour 25 minutes, over target. Fixes: replicate images to the recovery region, a break-glass login path independent of the primary region, and pre-provisioned application capacity at 10 percent. The second drill measures 22 minutes.

In the wild

Trade-offs

StrategyRPORTOCostChoose when
Backup and restoreHoursHours to a dayLowInternal or non-urgent systems
Pilot lightMinutesTens of minutesMediumImportant systems with tolerance for some downtime
Warm standbySeconds to minutesMinutesHigherRevenue-critical systems
Multi-site active-activeNear zeroNear zeroHighest, complexSystems where any downtime is unacceptable

Pitfalls in production

Leadership lens

Check yourself

  1. Nightly backups at 02:00, no log shipping. A disaster happens at 23:00. What is the data loss?

    Show answer

    About 21 hours: everything written since the 02:00 backup, which far exceeds most RPO targets for transactional data.

  2. Why can DNS failover take longer than the DNS change itself?

    Show answer

    Resolvers and clients cache records until the TTL expires, so some traffic keeps going to the old address for up to the TTL.

  3. Why keep backups in a separate account with immutability?

    Show answer

    So that compromised or mistaken production credentials, including those used by agents, cannot delete or encrypt them.

  4. What does a restore drill measure that a backup job's success status does not?

    Show answer

    Whether the backup can actually be restored into a working system, and how long it takes end to end, including dependencies and human steps.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →