Storing blobs in object storage
Open in the interactive app: practice, quiz, mock interview →In one minute
- Object storage keeps immutable blobs (objects) addressed by a bucket and key, with metadata, through a simple HTTP API: PUT, GET, DELETE, LIST.
- There are no real directories, no in-place updates, and in the standard classes no appends or renames. You replace whole objects, which lets the system scale to trillions of objects and exabytes.
- Internally it separates a metadata index (key to object location) from the data layer, which stores chunks with replication or erasure coding across failure domains.
- Amazon S3 is designed for 11 nines of durability and has offered strong read-after-write consistency since 2020; throughput comes from parallelism, not from fast single requests.
- It is the default home for backups, media, logs, data lakes, and model artifacts, because it is cheap per GB and needs no capacity planning.
How it works
The API model
A bucket is a namespace; a key is any string, like logs/2026/10/15/app-01.gz. The slashes are only characters. Tools display folders by listing with a prefix and a delimiter, which groups keys into "common prefixes." An object is written in full by one PUT (up to 5 GB), or by a multipart upload that sends parts in parallel (5 MiB to 5 GiB each, the last part smaller, up to 10,000 parts) and commits them atomically with a final call (S3 limits). Until the commit, readers see the old object or nothing. The largest S3 object was 5 TB for most of the service's life; since December 2025 it is 50 TB, about 48.8 TiB (AWS announcement), so 10,000 parts now means parts of several GiB for the biggest objects.
Optional features layer on top: versioning (every PUT creates a new version; DELETE adds a delete marker), lifecycle rules (move to colder tiers after N days, expire old versions), object lock for write-once compliance, and event notifications to queues or functions.
Inside: three layers
Front end. Stateless HTTP servers authenticate requests (signed with HMAC keys), apply bucket policies, and route.
Metadata index. A distributed, sorted key-value store maps (bucket, key, version) to object metadata: size, checksum, storage class, and the list of data chunks with their locations. It is partitioned by key range, and partitions split as they get hot. This is why listing is a sorted range scan and why prefixes affect throughput.
Data layer. Large objects are cut into chunks and spread across many storage nodes. Hot storage classes use replication or wide erasure codes across at least three availability zones; cold classes use more aggressive coding and slower media. Background processes continuously scrub data by checksum and repair lost fragments.
Consistency
A PUT returns only after the data is durable and the metadata write committed. With strong consistency, a GET or LIST immediately after a successful PUT sees the new object. There is no multi-object transaction, and concurrent PUTs to the same key resolve by last writer wins. Conditional writes (If-Match on ETag, If-None-Match: *) on PutObject, CompleteMultipartUpload and CopyObject, plus conditional deletes, make compare-and-swap patterns possible (S3 conditional requests). S3 only gained create-only writes in August 2024 (AWS announcement), which is why older designs put a lock table in a database beside the bucket.
Performance model
AWS quotes roughly 100 to 200 milliseconds for small-object latency and for the first byte of larger ones (S3 performance), and a single connection moves tens of MB/s. Throughput comes from many parallel requests: byte-range GETs for large objects and many concurrent PUTs, aligned to part boundaries where possible. Each prefix partition supports thousands of requests per second (S3 documents at least 3,500 writes and 5,500 reads per second per partitioned prefix), and the system adds partitions gradually as load grows, returning 503 Slow Down while it does.
The zonal exception
Newer single-zone classes relax the rules for latency-sensitive work. S3 Express One Zone stores data in one availability zone, in directory buckets with session-based auth, and targets single-digit millisecond access (S3 Express One Zone). It also breaks two classic rules: a PUT with x-amz-write-offset-bytes appends to an existing object, up to 10,000 appends (append docs), and RenameObject (June 2025) renames atomically in milliseconds without moving data (rename docs). The price is durability scope: nothing is copied to another zone, so a zone outage makes the data unavailable and a destroyed zone loses it. Treat it like a fast cache or a scratch tier, not the system of record.
Down the stack
Layer 1The contract
S3 promises strong read-after-write consistency for PUTs and DELETEs of individual objects, including LIST results and metadata changes, since December 2020 (S3 consistency). A successful PUT means the whole object is stored, never a partial one. Durability is stated as a design target of eleven nines per object per year, with objects stored across at least three availability zones in the standard classes, and availability as 99.99% design and a separate SLA (S3 data protection). These are different promises: an object can be temporarily unreachable without being lost. Integrity is checkable end to end: AWS's own clients now compute a checksum on every upload, CRC64NVME by default, the service recomputes it and rejects the upload on a mismatch, and other algorithms (CRC32C, SHA-256 and more) are available (S3 integrity). Concurrent writers get last-writer-wins unless they use conditional writes (If-None-Match: * to create only if absent, If-Match on an ETag to compare-and-swap).
Layer 2The mechanism
AWS has described parts of S3's internals publicly. Its storage nodes run ShardStore, a key-value store written in Rust and checked with lightweight formal methods that kept 16 bugs out of production (SOSP 2021), and objects are Reed-Solomon coded into k data and m parity shards (Warfield, 2023). It uses an LSM tree for its index and places shard data in large on-disk extents written sequentially, reclaiming space by copying live data out of an extent and resetting it. Open-source systems show the same pattern in more detail.
In Ceph's RADOS Gateway, an object becomes a head RADOS object holding metadata and the first part of the data, plus tail objects in fixed stripes (4 MB by default). A bucket index, stored in the object map of index objects and sharded so that one bucket's listing does not hit a single key-value instance, maps names to heads. RADOS places objects with CRUSH, a deterministic hash over the cluster map, so clients compute locations without asking a directory, and BlueStore writes data directly to raw devices with metadata in an embedded RocksDB. MinIO groups drives into erasure sets, stores each object's metadata in an xl.meta file alongside its erasure-coded parts, and inlines very small objects into that metadata file to save an I/O.
Layer 3The machine
Bulk object storage runs mostly on large hard drives because cost per terabyte dominates. A hard drive delivers around 120 random operations per second, about the same as when S3 launched in 2006, while capacity has grown enormously (Warfield, 2023), so as drives grow, IOPS per terabyte falls. S3 calls the resulting problem heat: requests landing on one disk at once, which it fights by spreading each customer's objects over a very large number of drives. The design responses are visible everywhere: pack small objects into large extents written sequentially, keep metadata in an LSM tree on faster media or in memory, spread every object across many drives so one request's I/O is parallel, and smooth hot spots by spreading a bucket's keys across many partitions.
On the network side, a single TCP stream's throughput is bounded by its congestion window divided by round-trip time, and a request also pays TLS and authentication. That is why the SDKs split large transfers into many parallel ranged GETs and multipart PUTs over several connections. Time to first byte sums front-end routing, a metadata lookup, and a hard-drive read, hence tens to hundreds of milliseconds. Hitting single digits, as the zonal classes do, means removing hops: one zone, co-located compute, and a session credential instead of signing every request from scratch.
Why it's built this way
- Storage cost per byte decides the economics, so data sits on dense hard drives behind erasure coding rather than replicas on SSD.
- Hard drives have few IOPS per terabyte, so writes are packed into large sequential extents and each object is spread over many spindles.
- A flat namespace without in-place updates or rename makes metadata a sorted key-value map that partitions cleanly by key range.
- Single streams and single drives are slow, so throughput comes from parallel requests, ranged reads, and multipart uploads.
- Corruption can occur anywhere along the path, so checksums travel with the data from client to disk and back, and background scrubbing repairs what decays.
Diagram
flowchart LR
C[Client] -->|"PUT /bucket/key"| FE["Front end (auth, routing)"]
FE --> META["Metadata index (sorted, partitioned)"]
FE --> SPLIT["Chunk + erasure code"]
SPLIT --> Z1["Storage nodes, AZ 1"]
SPLIT --> Z2["Storage nodes, AZ 2"]
SPLIT --> Z3["Storage nodes, AZ 3"]
META -->|"key -> chunk locations"| FE
SCRUB["Scrubber and repair"] --> Z1
SCRUB --> Z2
SCRUB --> Z3Worked example
A video platform stores 2 PB of original uploads and 6 PB of transcoded renditions, adding 40 TB per day. Each upload averages 1.5 GB.
Upload path. Clients get a pre-signed URL from the API and upload directly to the bucket with multipart, 64 MB parts, 8 in parallel. At around 20 MB/s per part stream, a 1.5 GB file finishes in about 10 seconds on a good connection, and a failed part is retried alone. The application servers never touch the bytes.
Cost. At roughly 0.02 USD per GB-month for standard storage, 8 PB costs about 160,000 USD per month. A lifecycle rule moves originals older than 90 days to an infrequent-access tier at around half the price, and renditions not watched in a year to an archive tier. That cuts storage spend by about 40%, at the price of retrieval fees and hours of delay for archived originals, which the business accepts.
Delivery. Viewers never read from the bucket directly. A CDN fronts it, so 95% of byte traffic is served from edge caches, and request costs and egress from object storage stay small.
Hot prefix. Transcoders initially wrote every output under renditions/ with keys starting by date. All writes hit one key range and got throttled (HTTP 503 SlowDown). Prefixing keys with a short hash of the video id spread writes across partitions.
In the wild
- AWS, 2017: a typo takes S3's index offline in US-East. While debugging the billing system, an engineer ran a capacity-removal command with a mistyped argument and removed far more servers than intended, including capacity behind the index subsystem (object metadata and locations) and the placement subsystem. Both needed a full restart, which had not been done in a large region for years, and S3 in us-east-1 was down for about four hours, taking EC2 launches, EBS restores, Lambda and AWS's own status page with it. AWS made the tool remove capacity slowly with minimum-capacity floors and sped up splitting the index into cells (AWS summary).
- Cloudflare, 2025: one abuse ticket disables all of R2. Handling a phishing report, an operator disabled the whole production R2 Gateway service instead of one bucket, because the admin tooling did not flag internal accounts. Every R2 operation failed for 59 minutes, Images, Stream and Vectorize failed with it, and recovery was slow because the tooling itself depended on R2; no stored data was lost. Cloudflare added Admin API guardrails for internal accounts and listed two-party approval for ad-hoc disables as in progress (Cloudflare postmortem).
- Microsoft Azure, 2014: a config flag loops the Blob front ends. A performance change meant for Table storage front ends was switched on across production at once, including Blob front ends, where a bug sent them into an infinite loop. Storage was intermittently unreachable in several regions for about ten hours, and some VMs took until two days later to recover. The flag was reverted within 30 minutes, but stuck front ends had to be restarted; Azure's deployment tooling now enforces staged rollout instead of relying on people (Azure RCA).
Trade-offs
| Option | Strengths | Weaknesses | Choose when |
|---|---|---|---|
| Object storage | Cheapest durable storage, no capacity planning, huge parallel throughput | High per-request latency, no in-place updates; appends and atomic rename only in zonal classes | Blobs, backups, data lakes, media |
| Block storage (EBS, persistent disks) | Low latency, random writes, any file system | Tied to one host and zone, capacity provisioned | Databases, boot volumes |
| Shared file storage (EFS, NFS) | POSIX semantics, many clients | Cost per GB higher, throughput limits | Legacy apps needing a shared file system |
| Self-hosted (Ceph RGW, MinIO) | Data on your hardware, no egress fees | You run durability and upgrades | Regulated or on-premises environments |
Pitfalls in production
- Treating keys as a file system. Renaming a "folder" means copying and deleting every object. Design keys so they never need renaming, and keep logical names in a database.
- Listing as a query engine. Listing millions of keys to find recent ones is slow and costly. Keep an index in a database or use inventory reports.
- Public buckets by accident. Misconfigured policies have caused many data leaks. Block public access at the account level and grant access via pre-signed URLs.
- Small objects at high volume. Request costs dominate when objects are a few KB. Batch small records into larger objects.
- Unbounded versions. Versioning without lifecycle rules keeps every overwrite forever. Expire noncurrent versions after a set period.
Leadership lens
- Model costs on four axes: storage per GB, requests, retrieval from cold tiers, and egress. Egress often surprises teams most.
- Define bucket ownership, encryption, and access policy as code, reviewed like any production change.
- Decide retention and lifecycle at design time, including legal holds, rather than after the bill grows.
- Use the object store's guarantees deliberately: conditional writes for coordination, events for pipelines, and multipart for anything over 100 MB.
Check yourself
Why can object storage scale further than a file system?
Show answer
Objects are immutable and addressed by flat keys, so there is no directory tree to lock, no in-place updates to coordinate, and metadata can be range-partitioned freely.
Why does a single GET feel slow while bulk transfers are fast?
Show answer
Each request pays tens to hundreds of milliseconds of latency, but many requests run in parallel across partitions and nodes, so aggregate throughput is very high.
What does a DELETE do in a versioned bucket?
Show answer
It adds a delete marker as the newest version, so GET returns not found while older versions remain retrievable until removed.
Why can keys that all start with today's date cause throttling?
Show answer
The metadata index is partitioned by key range, so sequential keys concentrate load on one partition until it splits.
Sources and further study
Primary
- Bornholt et al., Using lightweight formal methods to validate a key-value storage node in Amazon S3 (SOSP 2021): how S3 checks ShardStore against a model, and the bugs it caught.
- AWS: S3 maximum object size raised to 50 TB (2025): the current object size limit and the old one.
Systems docs
- S3: conditional requests: create-only writes, compare-and-swap and conditional deletes.
- S3: checking object integrity: the checksum algorithms and default client behaviour.
- S3 Express One Zone: directory buckets, session auth and single-zone durability.
- S3: data protection: durability, zone placement and which classes survive a zone loss.
Books
- System Design Interview, Volume 2, Alex Xu and Sahn Lam, Ch 9 "S3-like Object Storage": a whole-system interview walk-through of data store, metadata and versioning.
- Site Reliability Engineering, Beyer et al., Ch 26 "Data Integrity: What You Read Is What You Wrote": why durability needs backups, soft deletes and restore tests on top of replication.
Other curricula
- Azure Architecture Center: choose a storage service: frames Blob Storage as the default for unstructured data, with Data Lake Storage adding a hierarchical namespace for analytics.
- donnemartin/system-design-primer: mentions object storage only as advice to keep large blobs out of the database and store their location instead.
Watch
- How AWS S3 is built (The Pragmatic Engineer): an S3 leader on scale, durability practices, and consistency.
Free resources
- Building and operating a pretty big storage system called S3 (All Things Distributed): S3's scale, heat management, and engineering culture.
- Amazon S3 strong consistency: what read-after-write consistency means for S3 clients.
- S3: Best practices for performance: request rates per prefix, parallelism, and byte-range fetches.
- S3: Multipart upload overview: limits and lifecycle of multipart uploads.
- f4: Facebook's warm BLOB storage (OSDI 2014): moving warm blobs from replication to erasure coding.
- Finding a needle in Haystack (OSDI 2010): packing billions of small photos into large files.