Engineering Atlas › System Design Fundamentals

Raw volumes with block storage

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The interface

A 1 TiB NVMe drive with 4 KiB logical blocks is an array of 268,435,456 blocks. The host submits commands such as "read 8 blocks at LBA 1,048,576 into this buffer", and a completion arrives later. Block 77 is just 4,096 bytes with no name. A file system (ext4, XFS) keeps its superblock, inodes, bitmaps and journal inside those blocks, and databases on top think in 8 KiB or 16 KiB pages that line up with them.

Flush (or a FUA write) makes previously acknowledged writes durable; it is what fsync ultimately triggers. Discard (TRIM, UNMAP, NVMe Deallocate) says a range holds no data, so an SSD can erase it and a thin volume can release the space.

Block, file and object

File storage (NFS, SMB, EFS) moves the file system to a server shared by many clients. Object storage (its own lesson) drops in-place updates: whole objects by key over HTTP. Block storage is the lowest and fastest of the three, usually attached to one host at a time, because two hosts mounting one ext4 file system would corrupt each other's caches and allocation maps.

Where block storage lives

Local disks give the lowest latency, roughly 80 to 100 microseconds for a 4 KiB random read on a datacenter NVMe SSD, but die with the host. SANs put disk arrays on a dedicated network and present LUNs to servers over iSCSI or Fibre Channel. Cloud volumes are SANs at fleet scale: replicated extents on storage servers that a card or hypervisor presents to the guest as a local NVMe device, which you can detach, reattach elsewhere, resize and snapshot.

Snapshots and thin provisioning

A cloud volume is a map from logical block to physical location, and a snapshot freezes that map. The first write to a block afterwards goes to a new physical block, leaving the old one to the snapshot: copy-on-write, so a snapshot costs nothing up front and grows only with changes. AWS stores EBS snapshots incrementally in S3, replicated across the Region's zones (EBS snapshots), so a daily snapshot of a 1 TiB volume that changes 20 GiB a day costs about 20 GiB per day, not 1 TiB. Thin provisioning uses the same map: a 1 TiB volume consumes physical space only for blocks ever written, and discard returns it.

Down the stack

Layer 1The contract

A write that has completed may still sit in a volatile device cache; only after a completed flush, or a write with Force Unit Access, must it survive power loss. Writes are atomic only up to the device's atomic write unit, typically one logical block (512 B or 4 KiB). A 16 KiB InnoDB page written as four 4 KiB blocks can be torn: half old, half new after a crash. There is no ordering promise between concurrently outstanding writes, so a file system that needs "journal before data" must wait for completion and flush in between. Cloud volumes add availability and durability numbers: AWS states 99.8% to 99.9% annual durability for gp3 and 99.999% for io2 Block Express (EBS volume types), and performance limits, such as gp3's baseline of 3,000 IOPS and 125 MiB/s, provisionable up to 80,000 IOPS and 2,000 MiB/s at 500 IOPS per GiB (gp3 docs). Those gp3 ceilings were 16 TiB, 16,000 IOPS and 1,000 MiB/s until September 2025 (AWS announcement), so older sizing guides that stripe several gp3 volumes for performance may no longer need to. Note the durability figure is per volume, not per byte: a 0.1% to 0.2% annual failure rate means one or two lost volumes per 1,000 a year, which is why snapshots are not optional.

Layer 2The mechanism

In Linux, a file system submits I/O as struct bio requests, which the multi-queue block layer (blk-mq) places on per-CPU software queues, optionally merges and schedules, and maps to hardware dispatch queues (blk-mq docs). NVMe allows up to 65,535 I/O queues, each holding up to about 64K outstanding commands (NVM Express FAQ), so every core can have its own queue without locks. The driver writes a 64-byte command into the submission queue, rings a doorbell register, and the device posts a 16-byte completion entry and raises an interrupt or is polled.

Network volumes add a translation layer. Ceph RBD stripes an image across RADOS objects, 4 MiB by default (rbd manual), placed on OSDs by the CRUSH algorithm and replicated three ways; the primary OSD acknowledges a write only after every replica has persisted it. Each OSD runs BlueStore, which writes straight to the raw device, keeps its block map in an embedded RocksDB with a write-ahead log, and checksums everything it stores; the older FileStore backend, which journaled into a local file system, is unsupported from the Reef release on (Ceph storage devices). Snapshots are tracked per object, so the first write to an object after a snapshot clones it. Linux device-mapper's thin provisioning (kernel docs) keeps a B-tree from virtual block to data block on a metadata device, with reference counts so snapshots can share blocks; a block whose count drops to zero returns to the pool. EBS keeps its volume configuration in a cell-based store called Physalia, described in an NSDI 2020 paper as "millions" of small databases rather than one (Brooker et al.), so that re-mirroring after a server failure does not depend on one central database.

Layer 3The machine

Every I/O on a network volume is a network round trip plus a replicated write. On AWS Nitro instances, a dedicated card presents the EBS volume to the guest as an NVMe device and speaks AWS's own transport to storage servers; AWS promises gp3 only single-digit millisecond latency, and designs io2 Block Express for an average under 500 microseconds on 16 KiB I/Os (gp3 docs), against roughly 0.1 ms for instance-store NVMe.

Latency and IOPS are linked by Little's law: IOPS = outstanding I/Os / latency. At 0.5 ms per I/O, a single-threaded synchronous writer (queue depth 1) gets at most 2,000 IOPS no matter what was provisioned; reaching 16,000 IOPS needs at least 8 I/Os in flight. Throughput limits bind with large I/Os instead: at 125 MiB/s, 256 KiB writes top out near 500 per second. A database fsync on a network volume therefore costs about a millisecond, which is why commit-heavy workloads lean on group commit.

Databases guard against torn pages themselves: Postgres logs a full page image on the first change after a checkpoint (full_page_writes, Postgres docs), and InnoDB writes pages to a doublewrite buffer first, on by default (MySQL docs). Both spend write bandwidth that a larger atomic write unit would save, and that unit now exists in places. EBS offers torn write prevention for aligned 4, 8 and 16 KiB writes on every volume attached to a Nitro instance (EC2 docs), and Linux 6.11 added the RWF_ATOMIC flag so an application can ask for an all-or-nothing write: O_DIRECT only, a power-of-two length within the limits statx reports, at a naturally aligned offset (pwritev2(2)). An end-to-end atomic 16 KiB write is what makes it safe to switch off the doublewrite buffer; without one, switching it off trades integrity for speed.

Why it's built this way

  1. Disks and flash read and write in fixed units (512 B sectors, 4 KiB to 16 KiB flash pages), so the interface is fixed-size blocks, not bytes.
  2. Different software wants different structures on top (file systems, databases, VM images), so the device offers only an address space and leaves meaning to the layer above.
  3. Device caches are volatile and reordering raises throughput, so durability and order are opt-in through flush and FUA, not free on every write.
  4. A local disk dies with its host, so cloud volumes replicate over the network and accept extra latency in exchange for durability and detach-and-reattach.
  5. Copying a full volume to back it up is slow and expensive, so volumes keep a block map and snapshot by freezing it, paying only for changed blocks.

Diagram

flowchart TD
  App["Database: write 8 KiB page"] --> FS["File system (ext4/XFS)"]
  FS --> BIO["bio to blk-mq per-CPU queue"]
  BIO --> NVQ["NVMe submission queue + doorbell"]
  NVQ --> Card["Virtual NVMe (Nitro card / hypervisor)"]
  Card --> Map{"Block map: LBA to extent"}
  Map -->|"block shared with snapshot"| New["Allocate new extent (copy-on-write)"]
  Map -->|"block exclusive"| Same["Overwrite in place"]
  New --> Rep["Replicate to storage servers"]
  Same --> Rep
  Rep --> Ack["Completion entry to guest"]

Worked example

A Postgres primary on a 2 TiB gp3 volume does 6,000 commits per second at peak, each touching a few 8 KiB pages, with a 300 GB working set.

Commits first. Each commit needs a WAL fsync. At about 1 ms per flush and queue depth 1, one flushing process could commit 1,000 times per second. Group commit lets one fsync cover every transaction that arrived meanwhile, so 6,000 commits per second need only a few hundred flushes per second. The volume never sees 6,000 flush IOPS; latency is the limit, not provisioned IOPS.

Then page I/O. Checkpoints write dirty pages: if 40 GB of pages are dirtied between checkpoints every 15 minutes, that is about 45 MB/s spread over the interval, comfortably below gp3's 125 MiB/s baseline. Cache misses on a 300 GB working set with 128 GB of RAM produce perhaps 4,000 random 8 KiB reads per second, above gp3's 3,000 IOPS baseline, so provision 8,000 IOPS for headroom. Provisioning IOPS and throughput separately from size is the point of gp3; on a volume type that scales IOPS with size you would be buying unused terabytes to get the IOPS.

Snapshots: with 60 GB of changed blocks a day and 30-day retention, storage is one full copy of the written blocks plus 1.8 TB of increments. A restored volume loads lazily from S3, block by block on first read, so a freshly restored replica is much slower until it has touched its working set. Two paid options remove the guesswork: fast snapshot restore, which delivers full performance at creation, or a provisioned initialization rate of 100 to 300 MiB/s; at 300 MiB/s, 1.2 TB of written blocks takes a little over an hour (EBS docs).

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Local NVMe (instance store)Lowest latency, highest IOPS, no network costLost when the host dies or stops; no snapshotsCaches, scratch, replicated stores that rebuild from peers
Cloud network volume (EBS, PD)Durable, detachable, snapshots, online resizeHigher latency; IOPS and throughput caps; single AZMost single-node databases and VM boot disks
SAN (iSCSI, FC, NVMe-oF)Shared arrays, enterprise features, central managementSpecialist network and team; array is a shared failure domainOn-premises virtualisation and databases
Distributed block (Ceph RBD)Scale-out, self-healing, open sourceOperational complexity; tail latency under recoveryPrivate clouds, Kubernetes on premises
File storage (NFS, EFS)Many clients share files and directoriesWeaker semantics for databases, higher latencyShared content and home directories

Pitfalls in production

Leadership lens

Check yourself

  1. A gp3 volume is provisioned for 10,000 IOPS, but a single-threaded loader sees 1,200. Why?

    Show answer

    At queue depth 1, IOPS is capped by latency: about 0.8 ms per I/O gives roughly 1,200 per second. More parallel I/O, not more provisioned IOPS, is the fix.

  2. Why does deleting a week-old snapshot often free less space than its size suggests?

    Show answer

    Snapshots share unchanged blocks. Only blocks referenced by no other snapshot or by the live volume are freed.

  3. Why does Postgres write full page images to the WAL after each checkpoint?

    Show answer

    An 8 KiB page spans two 4 KiB blocks and can be torn by a crash. The full image in the WAL lets recovery restore a whole page before replaying changes.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →