Engineering Atlas › System Design Fundamentals

Why transactions across services are hard

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

What we are asking for

A local transaction gets atomicity almost for free: one forced commit record in one log decides the outcome. A distributed transaction has no single log. Several resource managers each own a slice of the change, each can crash independently, and the network can delay any message. Two-phase commit (its own lesson) answers with a promise followed by a decision. This lesson is about why that answer, and any answer like it, is expensive in ways that show up only under load, during failures, or when participants belong to different teams.

This is the multi-step consistency and partial failure problem shape. An AI agent that books a flight and a hotel through two vendor APIs faces the same question, except that neither API offers a prepare step, so the agent is in saga territory from the start.

Problem 1: participants block while in doubt

A participant that votes yes is in doubt until the decision arrives: its changes are durable but invisible, and its locks stay held. Participants can try cooperative termination and ask each other. If any peer committed, the decision was commit. If any peer aborted or never voted, commit was impossible, so abort is safe. But if every reachable peer is also prepared, nobody knows whether the coordinator logged commit before it died, and all of them wait. That is exactly the state left when the coordinator dies between its commit record and its first COMMIT message, so it is common, not exotic.

Problem 2: availability multiplies down

A transaction that needs every participant is only as available as all of them together. Four participants at 99.95% give about 99.80%: roughly 17 hours of failed or stalled transactions a year instead of 4.4. The coordinator is one more participant.

Problem 3: latency and lock hold time stretch

Locks are held until the decision reaches each participant. A local update on a hot row might hold its lock for 1 ms; inside a cross-zone 2PC, two round trips and two forced writes make it 10 to 30 ms. Throughput on one row is roughly the inverse of the hold time: 1,000 updates per second at 1 ms, 40 at 25 ms. Tails compound too, because prepare waits for the slowest vote. If each participant is slow 1% of the time, a transaction across five hits a slow one about 4.9% of the time. Google measured the same shape in production: in F1, which runs 2PC across Paxos groups in Spanner, 2PC usually doubles commit latency, commits take 50 to 150 ms, and the protocol works well up to tens of participants but aborts and latency rise sharply with hundreds (F1, VLDB 2013).

Problem 4: isolation does not compose

Atomic commit decides whether the writes happen together, not what concurrent readers see. If each participant runs snapshot isolation with its own clock, a reader can see shard A after a transfer commits and shard B before it: money disappears for a moment. Global serializability needs strict two-phase locking everywhere or globally comparable timestamps (Spanner's TrueTime, CockroachDB's hybrid logical clocks). Distributed deadlocks also escape every local detector, because each node sees only half of the cycle, so systems fall back to lock timeouts.

Problem 5: heterogeneity and heuristic outcomes

XA lets a transaction manager drive any resource that implements xa_prepare, xa_commit, xa_rollback and xa_recover, but the guarantee is only as strong as the weakest driver. Many systems cannot join at all. Kafka's transactions stop at Kafka: KIP-939, accepted in 2024, would let a producer hold a prepared transaction for an external coordinator, but its tracking issue is still open and scheduled for a future 4.x release (KIP-939, KAFKA-15370). DynamoDB's TransactWriteItems takes up to 100 actions, all in one account and Region, with no external participants (AWS docs). A payment provider's HTTP API has no prepare at all. Worse, some resources let an administrator or a timeout decide an in-doubt transaction alone. XA calls this a heuristic outcome; Java's JTA reports it as HeuristicMixedException when some participants committed and others rolled back. Atomicity is gone and a human must reconcile.

Problem 6: coupling across teams

A distributed transaction ties the participants' schemas, availability and deploy schedules together. Pat Helland argues that systems which must scale almost without limit make the transaction boundary one entity (an order, an account) on one machine, and coordinate between entities with retried, idempotent messages (Helland, CIDR 2007). The application then owns the workflow the transaction manager used to hide.

Down the stack

Layer 1The contract

The atomic commitment problem, as stated by Bernstein, Hadzilacos and Goodman, asks for five properties: all processes that decide reach the same decision; a decision, once reached, is never reversed; commit is decided only if every process voted yes; if there are no failures and all vote yes, the decision is commit; and once failures are repaired, every process eventually decides. Nothing promises a decision while failures persist. Skeen showed in 1981 that a protocol is non-blocking only if no participant state is adjacent to both commit and abort, a test 2PC's prepared state fails (Skeen 1981); with Stonebraker he then showed that no commit protocol stays non-blocking when the network can partition. Gray and Lamport later framed the fix as replicating the decision with consensus: Paxos Commit uses 2F + 1 coordinators, keeps 2PC's stable-storage delay, and reduces to classic 2PC when F = 0 (Gray and Lamport). A product promising non-blocking atomic commit across independent systems either runs consensus underneath or relies on timeouts that can violate atomicity.

Layer 2The mechanism

MySQL's XA PREPARE and XA RECOVER mirror Postgres's PREPARE TRANSACTION and pg_prepared_xacts. A JTA transaction manager such as Narayana or Atomikos keeps its own decision log and, on restart, calls xa_recover on each resource to list in-doubt Xids and match them against that log. If the log is lost, the matching is impossible.

Distributed databases move the coordinator's state into a replicated record. CockroachDB keeps a transaction record in a Raft range; with parallel commits, on by default since 19.2, it is marked STAGING along with the list of in-flight writes, and any node that finds an abandoned staging transaction can check those writes and finish it. That cut cross-range commit latency roughly in half, from two consensus rounds to one (Cockroach Labs). Spanner runs 2PC between Paxos groups, so the coordinator is itself replicated, and adds commit wait on TrueTime so commit timestamps order reads globally.

Layer 3The machine

With presumed abort, a commit across n participants costs n + 1 forced log writes (each prepare plus the coordinator's commit record) and 4n messages. On NVMe an fsync takes roughly 20 to 200 µs, but in a replicated store each "force" is a majority write: 1 to 2 ms within a region, 60 to 150 ms across continents. Waiting is not free either. In Postgres a prepared transaction pins the oldest visible transaction id, so VACUUM cannot remove newer dead tuples and tables bloat, and its locks sit in the shared lock table where every request on those rows queues behind them. Left long enough, it can force a shutdown to prevent transaction id wraparound, which is why the docs advise leaving max_prepared_transactions at zero unless an external transaction manager closes prepared transactions promptly (Postgres docs).

Why it's built this way

  1. Machines fail independently, so no single log can decide for all of them; someone must collect promises and announce one decision.
  2. A promise must survive a crash, so it is forced to disk before it is sent, and the participant must hold locks until it learns the outcome.
  3. A single coordinator can die holding the only copy of the decision, so participants block; removing that block requires replicating the decision with consensus.
  4. Consensus and global timestamps are feasible inside one system run by one team, so distributed databases absorb the cost there.
  5. Across organisations, nobody controls every participant's recovery code or uptime, so designs shrink the transaction to one entity and accept asynchronous, compensating coordination between entities.

Diagram

sequenceDiagram
    participant C as Coordinator
    participant A as Orders DB
    participant B as Inventory DB
    participant R as Waiting request
    C->>A: PREPARE
    C->>B: PREPARE
    A-->>C: YES (locks held)
    B-->>C: YES (locks held)
    C->>C: force COMMIT record
    Note over C: crashes before sending
    A->>B: what did you hear?
    B-->>A: prepared, no decision
    R->>B: UPDATE sku 42
    Note over B,R: waits on in-doubt lock
    C->>C: restarts, reads log
    C->>A: COMMIT
    C->>B: COMMIT
    B-->>R: lock granted

Worked example

A checkout writes an order in the orders database and decrements stock in the inventory database through a JTA transaction manager inside the checkout service, with the databases in different zones.

Availability: orders, inventory and the coordinating checkout instance are each 99.95%. Together that is 0.9995 cubed, about 99.85%, or around 13 hours a year of failed or stalled checkouts, before counting the payment call.

Contention: a local stock decrement holds the SKU row lock for about 0.8 ms. Inside 2PC the lock is held through the prepare round trip, the coordinator's forced write and the commit message, about 12 ms with cross-zone hops. In a flash sale one SKU gets 400 checkouts per second. At 0.8 ms the row absorbs about 1,250 per second; at 12 ms it caps near 83, so the queue grows by over 300 requests a second.

Coordinator crash: a checkout pod is killed right after forcing 120 commit records and before sending them. The 120 transactions sit prepared in inventory, locking 30 hot SKUs. Recovery runs only once the replacement pod reattaches the old log volume, four minutes later. Every checkout touching those SKUs waits, and the retry storm afterwards exceeds the original load.

The redesign makes the order the transaction boundary: checkout inserts the order as PENDING with an outbox event in one local transaction, and inventory reserves stock in its own local transaction and emits StockReserved or OutOfStock. Locks last under 1 ms, the databases fail independently, and a crash leaves a pending order to resume rather than locks to wait on.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
XA / 2PC across servicesReal atomicity, familiar modelBlocks on coordinator loss, multiplies downtime, heuristic outcomesFew trusted resources, low contention, short outages tolerable
Distributed SQL (Spanner, CockroachDB)Atomic and serializable, coordinator replicatedHigher write latency, one vendor holds all the dataThe data can live in one system and needs strong invariants
Single-entity transactions plus messagesLocal commits, scales with entitiesWorkflow logic moves into the app, eventual consistencyHigh scale, many teams, natural entity boundaries
Saga with compensationsNo cross-service locks, survives partial outagesNo isolation, compensations must be designed and testedMulti-step business processes across services
Reservation (try, confirm, cancel)Bounded holds with expiry, explicit business semanticsEvery participant needs reserve and expiry APIsInventory, seats, credit holds

Pitfalls in production

Leadership lens

Check yourself

  1. Coordinator is down; participant A is prepared, participant B is unreachable, participant C has aborted. Can A safely decide?

    Show answer

    Yes, it can abort. C aborted, so C never voted yes or was told to abort, and the coordinator cannot have logged commit. One aborted or unvoted peer is enough.

  2. Why does adding a fifth participant to a transaction hurt latency more than the extra network hop suggests?

    Show answer

    The prepare phase waits for the slowest participant, so the transaction's latency is the maximum of five distributions. Each extra participant raises the chance of hitting someone's tail and lengthens lock hold times for all of them.

  3. A team says their XA setup is safe because the database rolls back prepared transactions after ten minutes. What is wrong?

    Show answer

    That is a heuristic decision. If the coordinator had logged commit and other participants committed, the outcome is now mixed, which is exactly what the protocol exists to prevent.

  4. Why do Spanner and CockroachDB not suffer from the classic coordinator blocking problem?

    Show answer

    The coordinator's decision lives in a consensus-replicated record, so another replica or node can read it and finish the transaction if the original coordinator fails.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →