Design an S3-style object store

In one minute

• Split metadata from data, and treat the index as the hard part. Bytes are immutable blobs on thousands of disks. The index from (bucket, key) to blob is small, but it carries the consistency guarantees and LIST semantics.
• Write data first, commit metadata last. A PUT stores its fragments, then flips one index entry in a replicated log. That single commit makes overwrites atomic and read-after-write consistent. Failed and overwritten blobs become orphans that a collector removes after a grace period.
• Range-partition the index. LIST needs keys in sorted order, so the index is split into key ranges, each a small consensus group, that split automatically when they grow large or busy.
• Durability is about correlated failures and repair speed. Erasure-code across availability zones, checksum every byte end to end, scrub continuously and spread repair across many disks. Real data loss comes from bugs, bad batches and operators, not independent drive failures.
• Small objects and hot data shape the storage layer. Pack small objects into large volumes before erasure coding, and spread data so no single drive carries the hot set.

Requirements

Functional

• Buckets with globally unique names. PUT, GET (with byte ranges), HEAD and DELETE on very large objects, using multipart upload above about 100 MB. S3 raised its own ceiling from 5 TB to 50 TB in December 2025 (AWS); that is 10,000 parts of at most 5 GiB each (S3 multipart limits).
• LIST keys by prefix, with a delimiter that rolls deeper keys up into common prefixes, and pagination with a continuation token.
• Conditional writes: create only if absent (If-None-Match: *) and replace only if unchanged (If-Match: <etag>).
• Optional versioning, lifecycle rules (expire, or move to a colder tier after N days), and pre-signed URLs.

Non-functional

• Durability designed for 11 nines per object per year. Availability of 99.99% for reads and writes in a region. S3 states both numbers for its Standard class and stores objects across at least three AZs, so the loss of a whole AZ is survivable (S3 data protection).
• Strong consistency: once a PUT or DELETE returns, every later GET and LIST from any client sees it.
• First byte within 100 ms at p99 for small objects. Aggregate throughput scales with the number of keys and clients, not with any single server.
• Cost per stored GB close to raw disk cost, which rules out 3x replication for most data.

Out of scope

• Cross-region replication, event notifications, the IAM policy language and billing. They sit on top of the same core.

Back-of-envelope

Round planning numbers for one large region, not any provider's published figures.

• Objects and bytes today: 200 billion objects and 400 PB of logical data, a 2 MB mean. Most objects are under 100 KB; a few huge ones hold most of the bytes.
• Requests: 300 billion per day, about 3.5M per second on average and 7M at peak. Assume 70% GET or HEAD, 20% PUT and 10% LIST or DELETE, so about 1.4M PUTs per second at peak.
• Ingest: 1 PB of new logical data per day, about 11.6 GB/s or 93 Gbit/s on average. With 1.5x erasure-coding overhead the drives absorb 1.5 PB per day.
• Growth over 5 years: 400 PB + 5 × 365 PB ≈ 2.2 EB logical, 3.3 EB raw after encoding. On 20 TB drives filled to 70%, that is about 236,000 drives.
• Index size: about 500 bytes per object version (a 100-byte key, size, ETag, timestamps and a layout listing 12 fragment locations). 200 billion × 500 B = 100 TB, or 300 TB with three replicas, on SSD. It grows with object count, not bytes.
• Disk IOPS: a 20 TB hard drive delivers about 120 random IOPS (Warfield, All Things Distributed), the same as a 2 TB drive a decade ago, which is 6 IOPS per TB. 7M requests per second touching disk would need about 60,000 drives' worth of IOPS, so caching and spreading hot data are required, not optional.

API

PUT /{bucket}/{key}
  Content-Length: 5242880
  x-checksum-crc32c: "yZRlqg=="
  If-None-Match: *                         # optional: create only
  -> 200 ETag: "a1f3..."   412 Precondition Failed   400 BadDigest

GET /{bucket}/{key}
  Range: bytes=0-1048575                   # optional
  -> 206 ETag: "a1f3..."  Content-Range: bytes 0-1048575/5242880

GET /{bucket}?list-type=2&prefix=logs/2026/10/&delimiter=/&max-keys=1000
    &continuation-token=...
  -> 200 { "keys": [{"key": "logs/2026/10/16.gz", "size": 812345, "etag": "..."}],
           "common_prefixes": ["logs/2026/10/archive/"],
           "next_token": "bG9ncy8yMDI2LzEwLzE2Lmd6" }

DELETE /{bucket}/{key}
  If-Match: "a1f3..."                      # optional
  -> 204   412

POST /{bucket}/{key}?uploads               -> { "upload_id": "u-77" }
PUT  /{bucket}/{key}?uploadId=u-77&partNumber=3   -> ETag
POST /{bucket}/{key}?uploadId=u-77  { "parts": [{"n": 1, "etag": "..."}] }
  -> 200 ETag: "9c0e...-12"

The front end verifies the client's checksum before committing, so corruption in transit is caught at the door. Multipart rules are covered in the S3 lesson; a completed upload is simply a commit whose layout lists every part.

Data model

• buckets (a small, strongly consistent replicated database): name (primary key), owner, region, versioning, lifecycle_rules. Names are globally unique, so creation goes through one global registry.
• object_index (range-partitioned, each range a Raft group of three replicas on SSD): key (bucket, object_key, version_id descending), value {etag, size, created_at, storage_class, delete_marker, layout, user_metadata}. The key is ordered so LIST is a range scan and "latest version" is the first row for a key. Ranges are split by size (around 10 GB each) and by load, so a busy bucket spreads over many groups.
• layout (inside each index entry): the list of extents, each (volume_id, offset, length) for packed small objects or (stripe_id, fragment_ids) for large ones, plus the checksum of the whole object.
• placement_map (owned by a cluster manager, versioned, cached on every node): placement_group -> [12 drives across 3 AZs]. Changing it, for example after a drive dies, creates a new map version.
• volumes (on storage nodes): large append-only files of a few GB holding fragments or packed objects, each record with a header and a CRC32C. A per-volume index maps record ID to offset.
• gc_queue (a partitioned log): unreferenced blobs and when they became so.

High-level design

flowchart LR
  C["Client"] --> LB["Load balancer"]
  LB --> FE["Front end (auth, checksum)"]
  FE --> IX["Index ranges (Raft)"]
  FE --> PM["Placement map"]
  FE --> SN["Storage nodes (3 AZs)"]
  IX --> GQ["GC queue"]
  GQ --> GC["Collector"]
  GC --> SN
  SC["Scrubber and repair"] --> SN
  SC --> PM
  PMG["Partition manager"] --> IX

Write path. The front end authenticates, rate-limits and streams the body while computing its checksum. A small object (under about 1 MB) is appended to an open volume that is replicated three ways across AZs. A large object is cut into stripes, and each stripe is encoded 8+4 into 12 fragments written in parallel to the 12 drives of a placement group, 4 per AZ. When every fragment is acknowledged and the checksum matches the client's, the front end sends a commit to the index range that owns the key. The range leader checks any condition (If-None-Match, If-Match), appends the new entry to its Raft log, and replies once a majority has persisted it. Only then does the client get its 200. Any overwritten layout goes to the GC queue.

Read path. The front end asks the owning index range for the key; the leader answers under its lease, so the result reflects every acknowledged commit. With the layout in hand, it reads only the extents that cover the requested range, verifies each one's checksum and streams the bytes. If a data fragment is slow or missing, it reads parity fragments instead and decodes. LIST is a range scan from the prefix or continuation token, with delimiter roll-up done on the index server.

Deep dives

1. The commit protocol and the index

Why data first. If the index entry were written first, a reader could find a key whose bytes don't exist yet, or never will if the writer crashes. Writing fragments first and committing the index entry last means a reader sees the old version or the new one, never a mixture, and the commit is one small write to one consensus group however large the object is. The price is orphans: fragments from crashed uploads, abandoned multipart uploads and overwritten versions. A collector deletes an unreferenced blob only after a grace period of hours, which also protects readers that looked up the old layout just before an overwrite and are still streaming it. Deleting live data is the worst bug an object store can have, so the collector re-checks the index before every delete.

Index options. A hash-partitioned key-value store (in the style of DynamoDB) spreads load perfectly but cannot list keys in order without asking every partition. One database per bucket is simple but caps the size of a bucket. Range partitioning with a Raft group per range, as in Spanner, CockroachDB or TiKV, keeps keys sorted, gives linearizable conditional writes per key, and scales by splitting. I'd choose range partitioning. The known weak spot is sequential keys such as timestamps: every new key lands in the rightmost range, so splitting does not help write throughput. S3's documented guidance of at least 3,500 writes and 5,500 reads per second per prefix, scaling gradually beyond that with 503 Slow Down errors while it does, reflects the same pressure (S3 performance).

Strong consistency with caches. Reads go to the range leader, which uses a lease so it can answer without a round of consensus. Front ends may cache index entries for hot keys, but a cache that cannot tell whether it is stale breaks read-after-write. Each cached entry carries the range's commit index, which the front end checks against the leader's with a tiny request before serving it. AWS described the same idea when it made S3 strongly consistent in 2020: a new "witness" component is told of every object change and acts as a read barrier, so a cached entry is served only if it is not stale (All Things Distributed).

2. Durability: encoding, placement, scrubbing and repair

Encoding. Three-way replication costs 3x and survives 2 lost copies. An 8+4 Reed-Solomon code costs 1.5x and survives any 4 lost fragments. With 4 fragments per AZ, losing a whole AZ still leaves 8, enough to read every object. That leaves no margin during an AZ outage, which is acceptable because an AZ outage usually makes fragments unreachable rather than destroying them. A 9+6 code would add margin at 1.67x. Repair cost is the weakness of plain Reed-Solomon, which is why Azure Storage uses Local Reconstruction Codes: extra local parities let a single lost fragment be rebuilt from a small group instead of from 8 others (Huang et al., USENIX ATC 2012). The Erasure Coding lesson covers the maths. New small objects start in 3x-replicated open volumes, and a volume is erasure-coded once it is full and sealed, the same hot-to-warm path Facebook's f4 described (f4, OSDI 2014).

The nines. With a 1% annual drive failure rate (Backblaze measured 1.36% across about 344,000 drives in 2025, Backblaze) and 6 hours to repair, a fragment's drive fails within a given repair window with probability 0.01 × 6 / 8,760 ≈ 6.8 × 10^-6. Losing 5 of 12 independently in one window is about 792 × (6.8 × 10^-6)^5 ≈ 10^-23. That number is meaningless, because real risks are correlated: a firmware bug in one drive batch, a bad software release, or an operator deleting the wrong thing. The defences are mixing drive batches within placement groups, staged rollouts of storage software, and keeping deletes reversible for a while.

Checksums and scrubbing. Every record carries a CRC32C that is checked on write, on read and by a scrubber that reads every byte on every drive every few weeks. Disks return wrong data silently, and finding it early means repairing while the other 11 fragments are healthy. Checksums should start at the client: AWS-owned S3 clients now compute a checksum on every upload (CRC64NVME by default), and S3 recomputes and compares it before storing the object (S3 object integrity).

Repair speed. A dead 20 TB drive means rebuilding 20 TB of fragments, and each rebuilt fragment needs reads from 8 others: 160 TB of reads. One replacement drive writing at 250 MB/s would take 22 hours. Placement is therefore declustered: the dead drive's fragments belong to thousands of groups, so a thousand drives each rebuild a slice in parallel and repair takes under an hour.

3. Small objects, heat and the storage node

Small objects. Erasure-coding a 4 KB object into 12 fragments of a few hundred bytes would multiply IOPS by 12 and waste space on headers. Packing small objects into large volumes, the way Facebook's Haystack stored photos (Haystack, OSDI 2010), turns many small writes into sequential appends and makes each read one seek. Deletes leave holes, so a compactor rewrites volumes whose live fraction falls below a threshold, and the index entries move to the new location in the same commit protocol.

Heat. Drive capacity keeps growing while IOPS per drive stays flat, so the IOPS available per TB shrinks every generation. A freshly uploaded dataset is often read heavily for a day and then rarely touched. If its fragments sat on a handful of drives those drives would saturate, so placement chooses groups at random from the whole fleet, which spreads any one customer's hot data across thousands of spindles. Andy Warfield's FAST '23 talk describes heat management as a central problem at S3's scale. An SSD read cache absorbs most of the remaining hot reads.

Storage node software. It is deliberately simple (append, read by offset, verify, report health), because simplicity is a durability feature. AWS used lightweight formal methods to check that ShardStore, its storage node software, behaves like a simple reference model, including after crashes (SOSP 2021).

Down the stack

Failure modes

• 10x load on one prefix. A batch job hammers logs/2026/10/16/. The partition manager splits the hot range within minutes, and throttling (503 Slow Down, retried with backoff) protects other tenants meanwhile.
• AZ outage. Writes go to placement groups that avoid the failed AZ, reads decode from the 8 surviving fragments, and index ranges led from that AZ elect new leaders within seconds. Repair waits until the outage is declared long-lived, or the system would move petabytes for a 30-minute blip.
• Index range leader failure. Writes to that key range pause for one election timeout (about 1 to 3 seconds). Other ranges are unaffected, which is the point of having many small groups.
• Correlated drive failures. A firmware bug kills drives of one model. Mixing models within placement groups keeps each stripe's losses to one or two fragments, and per-model failure-rate alerts fire before repair falls behind.
• Collector bug. A bug marks live blobs as orphans. The grace period, the index re-check before deletion and a quarantine stage (moving, not erasing, for a few days) make it recoverable.
• Silent corruption. A bad NIC flips bits in transit. End-to-end checksums from the client's digest through to the disk record catch it, and the write fails instead of storing bad data.

In the wild

• AWS, 2017: a typo removes S3's index capacity. While debugging billing, an engineer ran a capacity-removal command with a mistyped argument and took out servers behind S3's index and placement subsystems in us-east-1. Both needed a full restart, which had not been done at that scale for years, so S3 in the region was down for about four hours and the status dashboard, which depended on S3, could not be updated. AWS made the tool remove capacity slowly with minimum-capacity checks and sped up splitting the index into smaller cells (AWS).
• Amazon S3, 2008: one flipped bit in gossip. A few internal server-state messages had a single bit corrupted, still parseable but wrong. S3 checksummed customer objects but not this gossip traffic, so the bad state spread until servers spent nearly all their time gossiping, and requests failed for about eight hours. AWS added checksums to the state messages and alarms on gossip rate, the same lesson as end-to-end checksums on data (AWS status archive).
• Cloudflare, 2025: an abuse action disables all of R2. Handling a phishing report, an operator disabled the whole R2 Gateway service instead of one bucket. Every R2 operation failed for 59 minutes, and recovery was slowed because the admin tooling itself depended on R2. No stored data was lost. Cloudflare added Admin API guardrails against disabling services in internal accounts and planned manager approval for ad-hoc disables (Cloudflare).

Trade-offs

Leadership lens

• Build first. The index with conditional commits, replicated storage with checksums, and the collector. Erasure coding is a cost optimisation you can add once the data path is proven correct.
• Rollout. Start with internal tenants whose data also lives elsewhere. Enable erasure coding for sealed cold volumes first, measure repair times, then widen. Ship storage software one AZ at a time.
• Ownership. Teams for the API, the index and the storage nodes, plus a durability team that owns scrubbing, repair and the collector and can block launches.
• Cost drivers. Drives, power and space dominate, so encoding ratio and fill level are the main levers, and lifecycle rules move the long tail to denser, slower media.
• Durability reviews. S3 runs a durability review, borrowed from security threat modelling, for any change that could affect durability, and prefers broad guardrails over point fixes. Copy that process before you copy the code.
• Design review questions. What is our repair time for one drive and for one rack, and how does it change as drives grow? Which single bug could delete live data, and what stops it?

Check yourself

1. Why does a PUT write data fragments first and commit the index entry last, and what cost does that ordering create?

   Show answer

   Committing the index first could let a reader find a key whose bytes do not exist yet, or never will if the writer crashes. Data first and one small index commit last makes overwrites atomic, but it leaves orphaned blobs that a cautious collector deletes only after a grace period and an index re-check.

1. Why is the index range-partitioned rather than hash-partitioned, and what is the known weak spot of that choice?

   Show answer

   LIST needs keys in sorted order, and a hash-partitioned index would have to ask every partition. Range partitioning with a Raft group per range keeps keys ordered and gives conditional writes per key, but sequential keys such as timestamps all land in the rightmost range, so splitting does not help their write throughput.

1. Using the lesson's numbers, why does a dead 20 TB drive force declustered placement?

   Show answer

   Rebuilding 20 TB of fragments needs reads from 8 others, about 160 TB, and one replacement drive writing at 250 MB/s would take about 22 hours. Declustering spreads the dead drive's fragments across thousands of placement groups, so a thousand drives each rebuild a slice and repair takes under an hour.

1. The independent-failure math gives data loss odds around 10^-23. Why does the lesson call that meaningless, and what defends against the real risk?

   Show answer

   Real losses come from correlated faults such as a firmware bug in one drive batch, a bad software release or an operator error. The defences are mixing drive models within placement groups, staged rollouts of storage software, and keeping deletes reversible for a while through grace periods and quarantine.

Sources and further study

Primary

• Erasure Coding in Windows Azure Storage (USENIX ATC 2012): Local Reconstruction Codes, which cut the fragments read during a repair.
• Backblaze Drive Stats for 2025: real annual failure rates by drive model, the input to any durability model.

Systems docs

• Amazon S3: conditional writes: If-None-Match, If-Match, the 412 and 409 cases and how multipart uploads interact.
• Amazon S3: checking object integrity: the supported checksum algorithms and the default CRC64NVME.
• Amazon S3: performance design patterns: per-prefix request rates, 503 Slow Down and parallelism.

Books

• System Design Interview, Volume 2, Alex Xu and Sahn Lam, chapter 9 "S3-like Object Storage": the metadata and data split, the data node design and multipart upload at interview depth.
• Designing Data-Intensive Applications (1st edition), Martin Kleppmann, chapter 6 "Partitioning": key-range versus hash partitioning, the trade-off behind the index choice.
• Database Internals, Alex Petrov, chapter 7 "Log-Structured Storage": the LSM trees used inside each index range and the append-only volume idea.

Other curricula

• Azure Architecture Center: Valet Key pattern: frames pre-signed URLs as a way to let clients move bytes straight to storage without passing through your servers.
• donnemartin/system-design-primer: treats object stores only as a building block behind CDNs, with no design of the store itself.

Watch

• FAST '23: Building and Operating a Pretty Big Storage System (USENIX): Andy Warfield on S3's drives, heat management, durability culture and ShardStore.
• How AWS S3 is built (The Pragmatic Engineer): an S3 leader on the architecture, the move to strong consistency and how the team works.

Free resources

• Diving Deep on S3 Consistency (All Things Distributed): how S3 added strong consistency without giving up performance.
• Building and operating a pretty big storage system called S3 (All Things Distributed): drives, heat, durability reviews and the people side of running S3.
• Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3 (SOSP 2021): how AWS checks ShardStore against a reference model.
• Finding a Needle in Haystack (OSDI 2010): packing small objects into large volumes with an in-memory index.
• f4: Facebook's Warm BLOB Storage System (OSDI 2014): moving sealed volumes from replication to erasure coding.
• CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data: Ceph's placement algorithm across failure domains.