Engineering Atlas › System Design Fundamentals

Detecting corrupted data with checksums

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The basic loop

The writer computes c = f(data) and stores or transmits (data, c). The reader recomputes f(data') and compares it with c'. A mismatch means reject, retry or repair. A match means probably intact: for an n-bit checksum and random corruption, a damaged block still matches with probability about 2^-n, one in 4.3 billion for 32 bits. For the error patterns hardware actually produces, good codes do far better than that and weak ones far worse.

The families

Additive sums. The internet checksum used by IPv4, TCP and UDP is the ones' complement of the ones' complement sum of 16-bit words (RFC 1071). Routers can update it incrementally when they decrement the TTL, but it cannot see two words swapped, because addition ignores order.

Position-weighted sums. Fletcher keeps two running sums: a adds each byte, b adds each new value of a, so position matters. Adler-32, used in the zlib format, is Fletcher with modulus 65,521 (the largest prime below 2^16) and a starting at 1.

Cyclic redundancy checks. A CRC treats the message as a polynomial over GF(2), where addition is XOR, and keeps the remainder after dividing by a fixed generator polynomial. The algebra gives guarantees, not just odds: a degree-n generator catches every burst of n or fewer flipped bits, and every pattern of scattered flips up to a limit that shrinks as messages grow. The IEEE 802.3 CRC-32 catches any 3 flipped bits in messages up to 91,607 bits (about 11 KB) but any 4 only up to 2,974 bits, while better 32-bit polynomials catch any 5 flips at Ethernet frame sizes (Koopman, DSN 2002). Ethernet, gzip, PNG and zip use CRC-32 (the IEEE 802.3 polynomial); iSCSI, SCTP, ext4, Btrfs, HDFS and Kafka use CRC-32C (the Castagnoli polynomial), which has better distance properties and dedicated CPU instructions.

Cryptographic hashes. SHA-256 is designed so that nobody can find two inputs with the same digest, even on purpose, which suits content addressing (Git objects, image digests, deduplication). It costs far more CPU than a hardware CRC, so storage engines use CRCs per block and reserve hashes for identity.

Where checksums live

A disk sector carries its own error-correcting code. An Ethernet frame carries a 4-byte CRC-32, checked and stripped at every hop. TCP adds its 16-bit checksum. A storage engine adds a CRC per page or block: RocksDB per block, Kafka per record batch, Postgres a 16-bit checksum per 8 KB page (on by default for new clusters since PostgreSQL 18, released September 2025; initdb --no-data-checksums turns it off) (PostgreSQL 18 release notes), HDFS a CRC-32C for every 512 bytes (dfs.bytes-per-checksum, dfs.checksum.type) (HDFS defaults). Kafka's record batch CRC is CRC-32C over every byte after the CRC field, placed so brokers can rewrite the offset without recomputing it (Kafka message format). Object stores add a whole-object checksum. S3 accepts about ten algorithms, from CRC-32C and CRC-64/NVME to SHA-256 and XXHash; AWS clients now compute a checksum on every upload by default, CRC-64/NVME unless told otherwise, and S3 recomputes it before storing (AWS).

The durability lesson uses a CRC to find the torn tail of a write-ahead log, and the key-value store lesson uses one per Bitcask record. This lesson is about which checksum to pick and where to put it.

The end-to-end argument

Saltzer, Reed and Clark argued in 1984 that a lower layer's check cannot stand in for the application's, because data can be damaged between layers. Ethernet's CRC is regenerated on every hop, so a switch with a bad memory cell forwards a corrupted payload under a fresh, valid CRC. Only TCP's weak 16-bit sum is left to catch it. Stone and Partridge measured real traffic in 2000 and found about 1 packet in 30,000 failing the TCP checksum on links whose CRCs should let through only 1 error in 4 billion: the damage happened inside routers and hosts (SIGCOMM 2000).

This is the trust-but-verify problem shape. It recurs in ML pipelines: a model file copied from training storage through caches to every inference node should carry the trainer's digest, verified at load time.

Down the stack

Layer 1The contract

A checksum promises one thing: if data and checksum disagree, something was damaged. A match only means no damage this code can detect, and that is spelled out per algorithm: CRC-32 detects all bursts up to 32 bits and all odd numbers of bit flips (its generator has x + 1 as a factor); Fletcher-16 cannot tell a byte of 0x00 from 0xFF because both are 0 mod 255; the internet checksum misses any reordering of 16-bit words. Nor does a checksum say which version is current. A lost write leaves an old block with a valid old checksum, so serious formats fold the block number and a generation counter into the checked bytes.

Layer 2The mechanism

Reflected CRC-32, as in zlib, starts with crc = 0xFFFFFFFF. For each byte it XORs the byte into the low 8 bits, then eight times shifts right and, if the bit shifted out was 1, XORs in 0xEDB88320 (the 802.3 polynomial bit-reversed). At the end it inverts all bits. The inversions make leading zero bytes change the result and make the empty string's CRC equal to zero. The standard check value is crc32("123456789") = 0xCBF43926; CRC-32C gives 0xE3069283 for the same input.

Implementations precompute a 256-entry table holding the effect of each byte, turning eight shift-and-XOR steps into one lookup: crc = table[(crc ^ b) & 0xFF] ^ (crc >> 8). "Slicing-by-8" uses eight tables to consume 8 bytes per step. The CRC after chunk a can seed the computation over chunk b, so a stream is checksummed as it flows, and zlib's crc32_combine joins CRCs computed in parallel.

Adler-32 is weak on short messages: under about 257 bytes, a never wraps the modulus, so the output uses only a fraction of its 32 bits. RFC 3309 replaced it with CRC-32C in SCTP for that reason.

Layer 3The machine

x86 CPUs since SSE4.2 have a crc32 instruction that consumes 8 bytes of CRC-32C with a 3-cycle latency and one-per-cycle throughput. Interleaving three streams reaches well over 10 GB/s per core, faster than an NVMe drive. It only knows the Castagnoli polynomial, a big reason new formats chose it; other polynomials use PCLMULQDQ (carry-less multiply) folding. ARMv8 has instructions for both CRC-32 and CRC-32C.

Corruption has physical sources. DRAM without ECC flips bits. NICs with checksum offload compute TCP checksums after data leaves host memory, so a bit flipped in a kernel buffer gets a valid checksum. Disks return the wrong sector, tear 4 KB writes on power loss, or acknowledge writes they never perform. SCSI T10 Protection Information adds 8 bytes per 512-byte sector: a 2-byte CRC guard, a 2-byte application tag and a 4-byte reference tag holding the expected sector number, catching misdirected I/O in hardware. Large operators also report CPU cores that silently compute wrong answers: Google sees a few such "mercurial" cores per several thousand machines (Hochschild et al., HotOS 2021), so verify on a different machine where it matters. A checksum computed on the faulty core over already-wrong data will match, which is why the check must be made by a different machine or at the far end.

Why it's built this way

  1. Every component that stores or moves bits flips a few, so data carries a short fingerprint that is cheap to recheck.
  2. Checks run on every read at line rate, so the fingerprint must cost about a memory copy, which favours CRCs over cryptographic hashes.
  3. Real faults come as bursts and a few scattered bits, so polynomial codes with proven burst and distance guarantees beat sums of the same width.
  4. CPUs gained an instruction for one polynomial, so storage built after 2008 standardised on CRC-32C.
  5. Per-layer checks leave gaps between layers, so the end that cares adds its own check over the full path.

Diagram

flowchart LR
  W["Writer: crc32c(block)"] --> P["Page cache"]
  P --> N["NIC: TCP csum offload"]
  N --> S["Switch: verify, regen FCS"]
  S --> R["Replica NIC: verify"]
  R --> D["Disk: sector ECC"]
  D --> RD["Reader: recompute crc32c"]
  RD -->|"match"| OK["Serve bytes"]
  RD -->|"mismatch"| FIX["Read other replica, repair"]

Worked example

A blob store holds 2 PB of user data in 4 MB chunks, about 500 million chunks, with three copies of each. Each chunk carries one CRC-32C per 64 KB sub-block (64 per chunk) plus a SHA-256 of the whole chunk recorded in metadata.

Why sub-blocks? With one CRC per 4 MB chunk, a 100 KB ranged read would have to fetch and check all 4 MB, a 40x amplification. With 64 KB sub-blocks it touches at most three, about 192 KB, for an overhead of 4 bytes per 64 KB.

CPU cost: at 50 GB/s of reads and 10 GB/s per core for hardware CRC-32C, verification needs about 5 cores fleet-wide. Software SHA-256 at around 500 MB/s per core would need 100, so SHA-256 runs only on writes and scrubs.

Detection odds: if hardware silently corrupts 1 chunk copy in 10 million per year, that is 150 a year across 1.5 billion copies. Each slips past a 32-bit CRC with odds of about 1 in 4.3 billion, so expected undetected corruptions are about 3.5 x 10^-8 a year. The checksum is not the weak spot.

Cold data is. A chunk nobody reads for years can quietly lose copies until none is good. Scrubbing all 6 PB every 14 days takes about 5 GB/s fleet-wide, a tenth of serving bandwidth, and each bad copy is repaired from a replica that matches the SHA-256 in metadata.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Internet checksum (16-bit)Trivial, updatable in placeMisses reordering, weak overallPacket headers that change per hop
Fletcher / Adler-32Very fast in software, order-sensitiveWeak on short messages, blind spotsLegacy formats, no CRC hardware
CRC-32CProven burst and distance guarantees, hardware speed32 bits only, no tamper resistancePer-block storage and log records
CRC-64Lower false match rate for huge objectsSlower without carry-less multiplyWhole-object checks on multi-GB files
SHA-256Collision resistant, content identitySeveral times the CPU of hardware CRCContent addressing, downloads, dedup
HMAC or signatureDetects deliberate tamperingNeeds key managementData crosses a trust boundary

Pitfalls in production

Leadership lens

Check yourself

  1. A switch with faulty buffer memory corrupts frames, yet no Ethernet CRC errors are reported. Why?

    Show answer

    The switch verifies the incoming CRC, corrupts the frame in memory, then computes a fresh CRC over the bad bytes when sending. Each link's CRC only covers its own wire.

  2. Why can't Fletcher-16 or the internet checksum distinguish certain corruptions that CRC-32 catches easily?

    Show answer

    Fletcher-16 works mod 255, so bytes 0x00 and 0xFF are equal to it, and the internet checksum is a commutative sum, so swapped words leave it unchanged. A CRC's polynomial division depends on every bit's position.

  3. A disk silently drops a write, leaving the previous version of the block in place. Will a per-block CRC stored in the block catch it?

    Show answer

    No. The old block and its old CRC are consistent. Catching lost writes needs the expected checksum or a generation number kept somewhere else, such as the parent pointer or an index.

  4. Why do new storage formats favour CRC-32C over the CRC-32 used by gzip?

    Show answer

    CRC-32C has better error detection for typical block sizes and has a dedicated x86 instruction (and an ARM one), so it runs at memory speed without lookup tables.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →