Concurrency and Race Conditions

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A race condition is when correctness depends on the timing of operations that run concurrently. It is the contention and ordering shapes at the scale of threads, processes or services.
The classic patterns: check-then-act (two threads both see "seat free" and both book it), read-modify-write (two increments, one lost), and unsafe publication (one thread sees a half-initialised object).
Races are timing-dependent, so they hide in tests and appear under load. Make them reproducible: stress tests, injected delays, and race detectors that reason about happens-before rather than luck.
Fix by removing the gap: atomic operations, locks held across the whole check and act, single-writer ownership, or database constraints and conditional updates.
Deadlocks are the opposite failure: too much locking in inconsistent order. Acquire locks in one global order and keep critical sections short.

How it works

Three patterns

Check-then-act. if seat.free: seat.book(user). Two threads both run the check before either acts; both book. The same bug appears in "create the user if the email does not exist", "send the welcome email if not already sent" and "start the job if no other worker has".

Read-modify-write. count = count + 1 is three steps: read, add, write. Two threads read 10, both write 11, and one increment is lost. Inventory counters, balances and rate limiters suffer from this.

Unsafe publication. One thread creates an object and stores a reference; another thread reads the reference and uses the object. Without a synchronisation point, compilers and CPUs may reorder writes, so the reader can see the reference before the object's fields are written. Language memory models define which operations create the ordering that makes publication safe. Go's model states that data-race-free programs behave as if operations were interleaved in some sequential order, and that programs with races are incorrect (Go memory model).

Happens-before

Two operations are ordered if one happens before the other: they are in the same thread in program order, or connected through synchronisation, such as a lock release followed by an acquire of the same lock, a channel send followed by the matching receive (Go memory model), or a thread start. If two accesses to the same variable, at least one a write, are not ordered by happens-before, that is a data race, whatever the timing on a particular run. This idea comes from Lamport's 1978 paper on logical clocks, and it is exactly what race detectors check.

Making races visible

Race detectors: Go's -race flag and ThreadSanitizer for C, C++ and others instrument memory accesses and track happens-before with vector clocks, reporting races even when the bad interleaving did not happen on that run. They only see code that actually executes, and they cost roughly 2 to 20 times in CPU and 5 to 10 times in memory (Go race detector), so they run in tests and staging under a realistic workload.
Stress tests: run the operation from 100 threads or 1,000 concurrent requests and check invariants (exactly 100 seats sold, final counter equals operations performed).
Injected delays: put a short sleep between the check and the act in a test build to widen the window from nanoseconds to milliseconds, which turns a one-in-a-million race into an every-time failure.
Database-level evidence: duplicate rows that a unique rule should have prevented, or counts that disagree with event logs.

Fixes

Remove the gap between check and act. Use atomic operations for counters (atomic.AddInt64, UPDATE stock SET qty = qty - 1 WHERE qty > 0). Hold a lock across the whole critical section, not just the write. Give each piece of state a single owner (one goroutine, one actor, one partition) and send it messages. In databases, use unique constraints, conditional updates or SELECT ... FOR UPDATE, and know which isolation level you run: Postgres's default, Read Committed, does not prevent lost updates in read-modify-write done in application code. An UPDATE that meets a concurrently updated row waits, then re-checks its WHERE clause against the new version, which is why putting the condition in the WHERE works. Repeatable Read instead aborts the second writer with SQLSTATE 40001, and the application must retry the whole transaction (Postgres docs).

Deadlocks

A deadlock needs a cycle: thread A holds lock 1 and waits for lock 2 while thread B holds lock 2 and waits for lock 1. Prevent it by always acquiring locks in a fixed global order (for example by account ID when transferring between two accounts), by holding locks briefly, and by using timeouts. Databases detect deadlock cycles and abort one transaction; Postgres checks only after a lock wait exceeds deadlock_timeout (1 second by default, because the check is expensive) and reports "deadlock detected" with both statements (Postgres docs); retry the transaction, as with serialization failures.

Down the stack

Layer 1The contract

A concurrent component promises that its invariants hold under any interleaving of its operations, not just the ones tests happened to produce. For shared data that means every access is either confined to one thread or ordered by synchronisation; for database state it means the invariant is enforced by a constraint, a conditional write or a lock, at a documented isolation level.

Layer 2The mechanism

Race detectors keep a vector clock per thread: a counter for every thread, recording how much of each other thread's history it has "seen" through synchronisation. Releasing a lock stores the releasing thread's clock in the lock; acquiring it merges that clock into the acquirer's. Each memory access is recorded with the accessing thread's clock. A new access races with a previous one if they come from different threads, at least one is a write, and neither clock is less than or equal to the other in every component. ThreadSanitizer keeps a compact shadow memory of recent accesses per 8-byte word to make this affordable.

Layer 3The machine

Races exist because hardware runs threads truly in parallel and reorders memory operations for speed. Each core has store buffers and caches; a write may sit in a store buffer before becoming visible to other cores, and x86 allows a later load to complete before an earlier store to a different address, while ARM allows even more reordering. Atomic instructions such as LOCK XADD or CMPXCHG on x86, or load-exclusive and store-exclusive pairs on ARM, make a read-modify-write indivisible by holding the cache line in exclusive state, at a cost of tens of nanoseconds when the line is contended across cores. Locks are built on these instructions plus memory barriers that force ordering, which is why correctly synchronised code is safe on every CPU and racy code may work on x86 and fail on ARM.

Why it's built this way

Parallel hardware and reordering make the order of memory effects unpredictable without synchronisation.
Correctness must therefore rest on happens-before, not on observed timing.
Testing samples only a few interleavings, so races hide.
Detectors that track happens-before find races that did not manifest on that run.
The robust fix removes the window entirely: atomic operations, single owners or enforced constraints.

Diagram

sequenceDiagram
  participant A as Thread A
  participant S as Shared seat
  participant B as Thread B
  A->>S: read free = true
  B->>S: read free = true
  A->>S: write owner = A
  B->>S: write owner = B
  Note over A,B: both believe they booked it
  Note over S: fix: one atomic conditional write

Worked example

A ticketing service sells 5,000 seats for a concert. In a load test with 20,000 concurrent buyers, it sells 5,047. The code checks SELECT status FROM seats WHERE id = $1, and if the seat is free, runs UPDATE seats SET status = 'sold', buyer = $2 WHERE id = $1. Both statements are inside a transaction at Read Committed isolation, which the team assumed made it safe.

It does not: at Read Committed, two transactions can both read "free" before either updates, then both update in turn; the second update simply overwrites the first. With 20,000 buyers competing for the most popular seats, 47 seats were double-sold. The time between check and act was only about 2 ms, but at 4 attempts per millisecond on the hottest seats, overlaps were common.

The fix collapses check and act into one statement: UPDATE seats SET status = 'sold', buyer = $2 WHERE id = $1 AND status = 'free', then check the affected row count; 0 means someone else won. The rerun sells exactly 5,000. The team adds the load test to CI with an invariant check (sold seats equals 5,000, no seat with two buyers in the payments table), and a reconciliation query in production that alerts on any seat with more than one successful payment.

In the wild

AWS DynamoDB, 2025: a stale check in DNS automation. Redundant DNS Enactors apply plans to Route 53. One was delayed, and its check that its plan was newer went stale; a second Enactor applied a newer plan and its cleanup then deleted the older plan, which the delayed Enactor had just made active. The regional endpoint was left with no IP addresses and manual repair was needed. DynamoDB errors lasted from 11:48 PM PDT on 19 October to 2:40 AM on 20 October, with downstream effects until about 2:20 PM. AWS disabled the automation worldwide pending a fix (AWS summary). It is check-then-act across machines.
Google Cloud, 2021: a race after validation. A race in the pipeline that pushes load-balancer configuration, in the code for about six months, could corrupt a config file after validation had already run. On 16 November it hit an unpatched cluster and external load balancers returned 404 for about half an hour until a rollback to the last good config; the incident record counts 1 hour 53 minutes in total. Google closed the race, added input validation, alerting and stronger pre-apply checks (Google Cloud incident report).

Trade-offs

Fix	Strengths	Weaknesses	Choose when
Atomic operation or conditional update	Fast, no lock to manage	Limited to simple operations	Counters, compare-and-set, single-row rules
Mutex around the critical section	General, easy to reason about	Contention, deadlock risk	In-process shared state
Single owner (actor, partition)	No shared state at all	Throughput per owner is limited	Hot entities, complex per-entity logic
Database constraint	Enforced for every writer	Error handling on conflict	Invariants that must never break
Serializable isolation	Strongest guarantees	Retries on serialization failures	Complex multi-row invariants

Pitfalls in production

Assuming a transaction prevents races. Fix: know your isolation level and use conditional writes or row locks.
Locking only the write, not the read that decided it. Fix: the lock or atomic operation must cover check and act together.
Inconsistent lock order across code paths. Fix: one global order, documented and enforced in review.
Tests that never run concurrently. Fix: stress tests with invariant checks and the race detector in CI.
"Fixing" a race with a sleep. Fix: a sleep changes timing, not ordering; add real synchronisation.

Leadership lens

Run race detectors in CI for languages that have them, and treat any report as a bug.
Ask in design review: which state is shared, who owns it, and what enforces each invariant under concurrency?
Prefer designs that avoid shared mutable state: single owners, immutable data, idempotent operations.
Watch for generated code that does check-then-act in application code; it is a very common pattern in agent output.

Check yourself

Why doesn't wrapping a check and an update in a Read Committed transaction prevent double booking?
Show answer
Read Committed lets both transactions read the row before either writes; nothing blocks the second reader, so both proceed and the second update overwrites the first.
What makes two memory accesses a data race in happens-before terms?
Show answer
They touch the same location from different threads, at least one is a write, and neither happens before the other through program order or synchronisation.
How do you prevent deadlocks when transferring money between two accounts?
Show answer
Always lock the two accounts in a fixed order, such as by account ID, so no cycle of waiting can form.
Why can racy code work on a developer's x86 laptop and fail on ARM servers?
Show answer
ARM allows more memory reordering than x86, so missing synchronisation that happened to be harmless on x86 becomes visible on ARM.

Sources and further study

Primary

AWS: DynamoDB service disruption, October 2025: the Planner and Enactor race in the provider's own words.
Google Cloud incident report, 16 November 2021: timeline of the pipeline race and its patches.

Systems docs

Lock Management settings (PostgreSQL): deadlock_timeout.

Books

Designing Data-Intensive Applications, 1st ed. (Kleppmann), Ch 7: Transactions, for lost updates, write skew and what each isolation level prevents.
Release It!, 2nd ed. (Nygard), Ch 4: Stability Antipatterns (Blocked Threads), for lock-related stalls in production.

Other curricula

The Jepsen consistency map (linked under Free resources) frames isolation levels as a hierarchy of which anomalies each one permits.

Free resources

Data Race Detector (Go): using -race, its overhead and how to read reports.
The Go Memory Model: what happens-before means for goroutines and channels.
ThreadSanitizer (Clang): race detection for C and C++.
Time, Clocks, and the Ordering of Events (Lamport): the original happens-before paper.
Transaction Isolation (PostgreSQL): what each isolation level does and does not prevent.
Consistency models (Jepsen): a map of isolation and consistency guarantees.

Open in the interactive app: practice, quiz, mock interview →