Engineering Atlas › System Design Fundamentals

Keeping a service up: availability and nines

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Defining and measuring it

The classic formula is time-based: uptime divided by total time. It hides a lot. A service that fails 2% of requests all month is "up" by a ping check and broken for users. Most modern teams use a request-based measure instead: successful requests divided by valid requests over a window, where "successful" includes a latency limit (a checkout that takes 30 seconds is not a success). Google's SRE practice calls this the service level indicator, and the target for it the SLO; Google measures most services by request success rate rather than uptime (SRE book, Ch 3).

The time-based view still explains what to improve. Availability equals MTBF / (MTBF + MTTR): mean time between failures over that plus mean time to repair. Repair time includes detection time (MTTD), so there are three levers: fail less often, notice sooner, and recover faster (AWS, Availability and Beyond). A service that fails once a month and takes 20 minutes to recover sits near 99.95%. Halving the recovery time buys as much as halving the failure rate.

Where you measure changes the number. Server logs miss requests that never arrived because DNS or the load balancer failed, so combine load balancer metrics with synthetic probes from outside.

The nines

AvailabilityDowntime per yearDowntime per 30 days
99%3.65 days7.2 hours
99.9%8.76 hours43.2 minutes
99.99%52.6 minutes4.3 minutes
99.999%5.3 minutes26 seconds

At four nines, a single incident that takes 5 minutes to notice and 10 minutes to fix spends more than three months of budget. That is why high targets force automation: humans cannot be paged, log in and act inside a 4-minute monthly budget.

Composition

If a request needs A and B and both must work, availability is A × B. Five dependencies at 99.9% each give about 99.5%, well below any one of them. If a request can be served by either of two independent replicas, availability is 1 − (1 − A)(1 − B): two 99% replicas give 99.99%. Quorum systems need k of n replicas up, which is a binomial sum. A 2-of-3 group of 99% nodes is up 99.97% of the time.

The word that matters is independent. Azure's guidance spells out what the parallel formula 1 − (1 − A)^N silently assumes: copies fail independently, hold identical or no state, lose no work in progress, and the survivors absorb the load without failing themselves (Azure Architecture Center). Replicas on the same rack, the same power feed, the same deploy pipeline or the same expired certificate fail together, and then the parallel formula is fiction. Real redundancy is placed across failure domains: hosts, racks, availability zones and regions, with deploys staggered so one bad release cannot reach all copies at once.

Redundancy patterns

Degrade instead of fail

A product page can render without recommendations or reviews. Timeouts, fallbacks and flags that switch off non-critical calls keep the core path up when a dependency is down. This is the partial failure shape: ask what still works when one part is not. A ride-hailing app shows a coarse ETA when the ETA service is down; an AI assistant answers without retrieval when the vector store times out, and says its answer is not grounded.

Availability also trades against consistency. During a network partition, a system that refuses writes on the minority side stays correct but unavailable there; one that accepts them stays available but must reconcile later. The Network Partitions and Consistency Models lessons go deeper.

Down the stack

Layer 1The contract

A usable availability contract names the indicator, the target, the window and the consequence: "99.95% of valid API requests in each calendar month return a non-5xx response within 1 second, measured at the load balancer; below that, customers receive a 10% service credit." Cloud SLAs usually exclude scheduled maintenance and customer-caused errors. The internal SLO is set tighter than the external SLA, so the team reacts before the contract is breached.

Layer 2The mechanism

Availability is delivered by detection and rerouting. An AWS Application Load Balancer target group checks every 30 seconds by default and marks a target unhealthy after 2 consecutive failures, so detection can take a minute. If every target fails at once, the ALB fails open and sends traffic to all of them, so a health check that breaks everywhere at once cannot by itself take the service down (AWS ALB docs). Envoy's passive outlier detection ejects a host after 5 consecutive 5xx responses for a base of 30 seconds, and caps ejection at 10% of hosts so a bad signal cannot empty the pool (Envoy docs). Route 53 marks an endpoint unhealthy once it fails a set number of consecutive checks and 18% or fewer of its checkers around the world still see it healthy, a rule meant to stop one isolated checker region from failing a healthy endpoint (Route 53 docs); clients must then expire their cached answer. For databases, Patroni holds a leader key in etcd with a TTL (30 seconds by default, renewed every 10-second loop) (Patroni docs); if the primary stops renewing it, a replica takes the key and promotes itself. VRRP (keepalived) moves a virtual IP to a backup router after about three missed one-second advertisements, plus a small skew that depends on priority (RFC 5798).

Layer 3The machine

The slowest detector is often the kernel. An established TCP connection to a peer that vanished, with no data in flight and default keepalive settings, waits 7,200 seconds before the first keepalive probe. With data in flight, Linux retransmits with exponential backoff and the default tcp_retries2 = 15 gives up only after at least 924.6 seconds, about 15 minutes (kernel docs). A client without its own timeout can therefore hang far longer than any failover takes. Moving a virtual IP relies on gratuitous ARP so switches update their tables within milliseconds. DNS failover is bounded by TTLs, and some resolvers and runtimes cache answers longer than the TTL says. Finally, a standby that has never served traffic has cold caches: a database whose 200 GB working set lives in the page cache can be up but serve 10x slower for minutes after promotion.

Why it's built this way

  1. Hardware, software and operators all fail, so no single component can be trusted for long enough to promise four nines.
  2. Redundancy only helps if copies fail independently, so copies are spread across failure domains.
  3. Redundant copies are useless until traffic moves, so detection and rerouting are automated with health checks, leases and virtual IPs.
  4. Detection too eager causes needless failovers and flapping, so checks use consecutive-failure thresholds and ejection caps, which trades a little detection time for stability.
  5. Each nine costs roughly an order of magnitude more effort, so targets are set by what users and the business need, not by what sounds impressive.

Diagram

sequenceDiagram
    participant U as User
    participant LB as Load balancer
    participant A as App in zone A
    participant B as App in zone B
    LB->>A: health check
    A--xLB: timeout
    LB->>A: health check
    A--xLB: timeout
    Note over LB: 2 failures, mark A unhealthy
    U->>LB: request
    LB->>B: route to healthy target
    B-->>LB: 200 OK
    LB-->>U: 200 OK

Worked example

A checkout path calls a CDN (99.99%), a load balancer (99.99%), an app tier of three instances each up 99.5%, a Postgres primary with automatic failover (99.95%), and a third-party payment API (99.9%). The app tier in parallel is 1 − 0.005³, effectively 100%. The chain multiplies to about 99.83%, or 73 minutes of downtime a month. The payment API alone accounts for more than half of it.

Adding a fourth app instance changes nothing visible. Moving payment authorization off the synchronous path does: the checkout writes an order with status pending to a durable queue (99.99%) and confirms to the user, and a worker charges the card with retries. The chain becomes about 99.92%, or 35 minutes a month, and payment outages now delay confirmation emails instead of losing sales. The next nine would come from the database: multi-zone synchronous standbys and a tested failover under 30 seconds.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Single instance plus fast restoreCheap, simpleMinutes to hours of downtimeInternal tools, batch jobs
Active-passiveSimple consistency, one writerFailover gap, idle standbyStateful stores with a single writer
Active-active in one regionNo failover gap for stateless tiersNeeds N+1 capacityMost web and API tiers
Multi-region active-activeSurvives a region lossWrite conflicts, high cost and complexityRevenue-critical global services
Graceful degradationCore path survives dependency lossMore code paths to testMany optional dependencies

Pitfalls in production

Leadership lens

Check yourself

  1. A service depends on six internal services, each at 99.95%. What is its best possible availability, and what does that imply?

    Show answer

    About 0.9995⁶ ≈ 99.7%. It cannot promise 99.9% unless it removes dependencies from the path, makes some optional, or gets them to higher targets.

  2. Why can two replicas in the same availability zone give far less than the parallel formula predicts?

    Show answer

    The formula assumes independent failures. A zone outage, shared power or a shared bad deploy takes both down together.

  3. When does a 2-of-3 quorum have lower availability than a single node?

    Show answer

    When each node's availability is below 50%. Requiring a majority then makes the group less likely to be up than one node, which is why quorums need individually reliable members.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →