Engineering Atlas › System Design Interviews

Consistency across services in interviews

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why the problem exists

Inside one database, a transaction is atomic and isolated. Across services, each service commits independently. If inventory commits a reservation and then payment fails, someone must release the reservation. If the order service writes an order and crashes before publishing "order created", downstream services never hear about it.

Two-phase commit

A coordinator sends PREPARE to all participants. Each participant does the work, writes it durably with locks held, and votes yes or no. If all vote yes, the coordinator durably records "commit" and sends COMMIT; otherwise it sends ABORT. Once a participant votes yes, it cannot decide alone: it must wait for the coordinator, holding locks. If the coordinator crashes after collecting votes, prepared participants are stuck until it recovers. In practice operators are tempted to break the deadlock by hand, and XA names this: a resource manager may make a "heuristic" decision to commit or roll back a prepared branch on its own, which frees locks but can leave the global transaction half committed, so it must report the decision to the transaction manager (X/Open XA specification, 1991). Spanner runs 2PC over participants that are themselves Paxos-replicated groups; its authors say this "mitigates the availability problems", and they measured commits scaling reasonably to 50 participants, with latency rising noticeably at 100 (Spanner, OSDI 2012). Across independently owned services and third-party APIs, 2PC is usually impractical: a card network will not hold a prepared transaction for you.

Sagas

A saga is a sequence of local transactions T1, T2, ..., Tn with compensations C1, ..., Cn-1. If Tk fails, run Ck-1, ..., C1. The idea predates microservices: Garcia-Molina and Salem proposed it in 1987 for long-lived transactions inside one database, such as an airline booking that holds seat locks for hours, and noted that nothing tells other transactions that they saw work later compensated (Sagas, SIGMOD 1987). Compensations are semantic, not rollbacks: "refund the charge," "release the reservation," "cancel the booking," and they must be idempotent and retryable because they also run over unreliable networks.

There are two styles. Orchestration: a saga orchestrator (a service or a workflow engine such as Temporal or AWS Step Functions) holds the state machine, calls each step and decides what to do next. Choreography: each service listens for events and reacts, with no central controller. Orchestration is easier to monitor and change for longer sagas; choreography fits short flows but risks cyclic dependencies between services and is hard to integration-test (Azure Architecture Center).

Order steps so that the riskiest or least reversible step comes late. The pivot step is the point of no return (often the payment capture); steps before it are compensatable, steps after it should be retryable until they succeed.

Isolation in sagas

Sagas give up isolation. Between T1 and Tn, other transactions can see that stock is reserved for an order that will be cancelled. The anomalies are the familiar ones: lost updates, dirty reads of a step that is later compensated, and non-repeatable reads across steps. Countermeasures include semantic locks (mark the order PENDING so others treat it carefully), commutative updates, a pessimistic ordering that moves updates others might read into the retryable steps after the pivot, re-reading values before acting, and designing user-visible states for "in progress" (Azure Architecture Center).

Transactional outbox

A service writes its business change and an event row into an outbox table in the same local transaction. A relay (polling or change data capture such as Debezium reading the database log) publishes outbox rows to the message broker (microservices.io). This avoids the dual-write problem where the database commit succeeds and the publish fails, or vice versa. Consumers must be idempotent because the relay delivers at least once.

How it shows up in interviews

Triggers: design an e-commerce checkout, a travel booking system (flight plus hotel plus car), a payment or wallet transfer, a food-delivery order flow, a banking ledger across accounts in different shards. A strong answer explains why a distributed ACID transaction is unavailable or undesirable, chooses saga with orchestration, lists steps with compensations, places the pivot, uses the outbox for each step's events, makes steps idempotent, and states what intermediate states users can see. Probes: what if a compensation fails, what if the orchestrator crashes, what if two sagas touch the same inventory, when would you use 2PC instead.

Down the stack

Layer 1The contract

2PC promises atomicity (all participants commit or none do) and, with locks held through commit, isolation, at the cost of blocking during coordinator failure. A saga promises eventual all-or-compensated: given that every step and compensation is eventually retried to success, the system ends with either all steps applied or all applied steps compensated. It does not promise isolation; intermediate states are visible. The outbox promises that an event is published if and only if its local transaction committed, at least once.

Layer 2The mechanism

In Postgres, a participant runs PREPARE TRANSACTION 'gid', which persists the transaction's state and locks so it survives a crash, and later COMMIT PREPARED 'gid' or ROLLBACK PREPARED 'gid'; max_prepared_transactions defaults to 0, so it must be enabled (PostgreSQL docs). The docs are blunt that the command is for external transaction managers, not applications (PostgreSQL docs). XA defines the same xa_prepare, xa_commit and xa_rollback calls for resource managers. A saga orchestrator persists a state record per saga (saga_id, current_step, status such as RUNNING, COMPENSATING, COMPLETED, FAILED) before and after each step, and passes saga_id + step as the idempotency key to each service. Workflow engines like Temporal persist an event history for each workflow and replay it to rebuild state after a worker crash, so the orchestrator code looks sequential but survives failure. The price is determinism: replay must issue the same commands in the same order, so workflow code may not read the clock, draw random numbers or call services directly; those go in activities whose results are recorded in the history (Temporal docs). For the outbox, Debezium reads Postgres's logical replication stream from a replication slot, turns inserted outbox rows into Kafka records, and by default routes each to the topic outbox.event.<aggregatetype>, carrying the row's id as a header that consumers can use to drop duplicates (Debezium docs).

Layer 3The machine

Each durable decision costs a log flush. In 2PC, every participant flushes its prepare record (an fsync of the WAL) before voting yes, and the coordinator flushes its decision before sending commit, so the critical path contains at least two sequential flushes and two network round trips; within a data centre that is a few milliseconds, across regions it is hundreds. Locks held during that window block other transactions on the same rows, so throughput on hot rows drops to roughly one transaction per commit latency. Prepared transactions also pin resources: in Postgres a forgotten prepared transaction holds its locks and prevents vacuum from cleaning up old row versions, so table bloat grows until someone resolves it, and in the extreme the server shuts down to avoid transaction ID wraparound (PostgreSQL docs). In a saga, each step commits locally and releases locks immediately, so hot rows stay available; the cost moves to extra writes (saga state, outbox rows) and asynchronous latency between steps through the broker.

Why it's built this way

Network partitions and crashes are unavoidable, and a participant that has promised to commit cannot safely decide alone, so atomic commit must block or must replicate the coordinator. Holding locks across slow networks destroys throughput, so cross-service workflows release locks early and compensate instead. Writing to a database and a broker cannot be atomic across the two, so the event is written into the database and relayed. Every step can be retried, so every step and compensation must be idempotent.

Diagram

sequenceDiagram
    participant O as Saga orchestrator
    participant I as Inventory
    participant P as Payments
    participant S as Shipping
    O->>I: reserve stock (saga 7, step 1)
    I-->>O: reserved
    O->>P: charge card (saga 7, step 2)
    P-->>O: charged
    O->>S: book courier (saga 7, step 3)
    S-->>O: no courier available
    O->>P: refund (compensate step 2)
    P-->>O: refunded
    O->>I: release stock (compensate step 1)
    I-->>O: released
    O->>O: mark saga 7 COMPENSATED

Worked example

A travel site books a flight, a hotel and a rental car as one trip, 200 trips per minute at peak. Each supplier is an external API with its own booking and cancellation endpoints; none supports 2PC.

The orchestrator stores saga state in its own database. Step order: hotel hold (free cancellation, cheap to undo), car hold (free cancellation), then flight ticketing (expensive to cancel, so it is the pivot), then confirmation email (retry until success). Each call includes an idempotency key trip-{id}-step-{n}.

In a typical week, 3% of trips fail at the flight step because the fare changed. For those, the orchestrator cancels the car and the hotel. About 0.1% of cancellations fail on the first attempt (supplier timeouts); they retry with exponential backoff for up to 24 hours. Twice a month a cancellation still fails after a day, and the saga moves to NEEDS_ATTENTION, which pages the operations team with the supplier reference.

The orchestrator crashes during a deploy with 150 sagas in flight. On restart it loads every saga in RUNNING or COMPENSATING and resumes from the last recorded step. Because each supplier call is idempotent, re-sending a step that had already succeeded returns the existing booking rather than booking twice.

Users see a "Booking in progress" page for the 5 to 20 seconds the saga takes, then a confirmation or a clear failure.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
2PC or XAAtomic and isolatedBlocking, lock holding, needs participant supportInside one database system or tightly coupled stores
Orchestrated sagaClear flow, central monitoring, easy to changeOrchestrator is a component to run and scaleMulti-step business workflows
Choreographed sagaNo central service, loose couplingHard to trace and change, cyclic dependenciesShort flows of two or three steps
Try-confirm-cancelReservations with explicit expiryEvery participant must implement three operationsInventory, seats, quotas
Avoid distributionTrue ACIDLarger service or shared databaseData that must change together frequently

Pitfalls in production

Leadership lens

Check yourself

  1. Why can a participant in 2PC not decide on its own after voting yes?

    Show answer

    It promised to commit if asked, and other participants may already have committed or aborted based on the coordinator's decision, so acting alone could break atomicity.

  2. What is a pivot step in a saga, and where should it go?

    Show answer

    The point after which the saga will complete rather than compensate; put it after cheap, reversible steps so failures before it are easy to undo.

  3. What problem does the transactional outbox solve?

    Show answer

    It makes the business write and the event publication atomic by storing the event in the same database transaction and relaying it afterwards.

  4. Why must compensations be idempotent?

    Show answer

    They run over unreliable networks and are retried, so running one twice must not, for example, refund a payment twice.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →