Engineering Atlas › System Design Fundamentals

Long-running transactions with the saga pattern

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Local transactions and compensations

Each step is a normal ACID transaction inside one service: reserve stock, authorize payment, book a courier. Each step that can be followed by a failure needs a compensation: release stock, void the authorization, cancel the booking. A compensation is not a rollback. The original commit happened and may have been observed (an email sent, a hold visible on a card), so the compensation is a new business action that semantically reverses it. Garcia-Molina and Salem, who named the pattern in 1987, said exactly this: a compensating transaction undoes the step "from a semantic point of view" without necessarily restoring the earlier state. Their target was long-lived transactions inside one database, which held locks for hours; microservices later reused the idea across service boundaries (Garcia-Molina and Salem).

Pivot and retriable steps

Order the steps deliberately. Compensatable steps come first. Then a pivot step, the point of no return (for example, capturing payment). After the pivot, only retriable steps remain: steps that are guaranteed to succeed eventually if retried, such as sending a confirmation. This structure means a saga either compensates fully before the pivot or completes after it, never stranded half way. The original paper calls these two directions backward recovery (compensate) and forward recovery (retry from a save-point), and the Azure Architecture Center uses the compensable, pivot and retryable vocabulary (Azure).

Orchestration and choreography

In orchestration, a saga orchestrator stores the saga's state, sends a command to each service, waits for the reply, and decides the next step or the compensation path. The whole flow is in one place, easy to test and observe. In choreography, each service publishes an event when it finishes ("StockReserved") and the next service subscribes. There is no central component, but the flow is spread across services, and cycles or missing handlers are hard to see.

Delivery and idempotency

Commands and events travel through a broker with at-least-once delivery, so every step and compensation must be idempotent: keyed by saga id and step, and safe to repeat. Each service must also publish its reply atomically with its local commit, which is what the outbox pattern provides.

Isolation anomalies

Because each step commits immediately, others can see partial results. A customer might see stock reserved for an order that is about to be cancelled, or two sagas might read the same balance and both proceed. The standard catalogue names three anomalies: lost updates, dirty reads and fuzzy (non-repeatable) reads. Countermeasures include semantic locks (a PENDING status that other operations respect), commutative updates (increments rather than overwrites), re-reading values before the pivot, a pessimistic view (reorder steps so updates others might read happen in retryable steps, after the pivot), version files (record operations so out-of-order ones can be reordered), and choosing the mechanism by value, keeping a real distributed transaction for the few high-risk updates (Azure). Chris Richardson notes that picking and implementing these takes careful analysis per saga (microservices.io).

Down the stack

Layer 1The contract

A saga promises eventual all-or-nothing at the business level: either every step's effect is applied, or every committed step before the failure has been compensated. It does not promise isolation, so intermediate states are visible. Each step is executed at least once, never exactly once, so the contract to each service is "you may receive this command more than once; produce the same result". Compensations themselves must eventually succeed; if they cannot, the saga ends in a state that needs a human.

Layer 2The mechanism

An orchestrator keeps a saga instance row, for example saga_id, type, current_step, status (RUNNING, COMPENSATING, DONE, FAILED), and serialized data, updated in the same transaction that enqueues the next command into its outbox. Commands carry the saga_id as a correlation id; replies are matched back by it. Participants keep a processed_messages table keyed by message id to drop duplicates. Workflow engines package this: Temporal records an event history (WorkflowExecutionStarted, ActivityTaskScheduled, ActivityTaskCompleted, ActivityTaskFailed) and, after a worker crash, rebuilds workflow state by deterministically replaying the workflow code against that history (Temporal docs). AWS Step Functions expresses steps as states with Retry and Catch fields that route failures to compensation states. Its retry defaults are worth knowing: 3 attempts, a 1 second first interval, a backoff rate of 2.0, and no jitter unless you set JitterStrategy to FULL (AWS docs).

Layer 3The machine

Every step costs at least one local commit with a WAL fsync (50 µs to 1 ms on SSD) and one durable message. With Kafka and acks=all, the producer default since Kafka 3.0 along with idempotence (KIP-679), a publish waits for the partition leader and in-sync followers to persist the record, typically a few milliseconds in one region. A five-step saga therefore involves about ten commits and ten messages, so 50 to 200 ms end to end before any business logic, plus consumer poll intervals. Duplicates come from physical events: a consumer processes a message, then crashes before committing its offset, and after the rebalance the message is delivered again; an SQS message whose processing exceeds the visibility timeout (30 seconds by default, extendable up to 12 hours from first receipt) reappears to another consumer (AWS docs). Timeouts on a step must be longer than the slowest normal response, or the orchestrator compensates a step that actually succeeded.

Why it's built this way

  1. A distributed transaction would lock data across services and couple their availability, so each service commits locally.
  2. Local commits cannot be rolled back after the fact, so failures are handled with compensating business actions.
  3. Messages between services can be lost or duplicated, so delivery is at-least-once and every handler is idempotent.
  4. Compensations can themselves fail, so steps are ordered with a pivot, leaving only retriable work after it.
  5. A flow spread across many services is hard to reason about, so complex sagas centralize state in an orchestrator with a durable log.

Diagram

sequenceDiagram
    participant O as Orchestrator
    participant I as Inventory
    participant P as Payments
    participant S as Shipping
    O->>I: ReserveStock
    I-->>O: Reserved
    O->>P: AuthorizePayment
    P-->>O: Authorized
    O->>S: BookCourier
    S-->>O: Failed (no capacity)
    O->>P: VoidAuthorization
    P-->>O: Voided
    O->>I: ReleaseStock
    I-->>O: Released
    Note over O: saga status COMPENSATED

Worked example

An online store handles 300 orders per second at peak. The order saga has five steps: create order as PENDING (semantic lock), reserve stock, authorize payment, capture payment (the pivot), and request shipment (retriable).

About 2% of orders fail at payment authorization, so 6 sagas per second compensate: release stock, mark order REJECTED. Each step averages 15 ms, so a happy-path saga finishes in about 100 ms including messaging, and the orchestrator holds about 30 in-flight sagas at any moment (Little's law: 300 × 0.1 s).

A bad day: the payment provider's API slows to 20 seconds per call while the step timeout is 10 seconds. The orchestrator times out, assumes failure, and compensates by voiding an authorization that may still succeed. Without an idempotency key on the authorization call, a retry creates a second hold on the customer's card. The fixes: send a stable idempotency key (saga_id plus step) to the provider, query the provider's status before compensating, set the timeout above the provider's P99.9, and alert when sagas sit in COMPENSATING for more than a minute. A daily report lists sagas in FAILED for manual handling; at 300 orders per second, even a 0.01% stuck rate is 2,600 orders per day, so automation of common failures matters.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Orchestrated sagaFlow in one place, easy to observe and changeOrchestrator is a component to run and scaleMore than three steps, or complex branching
Choreographed sagaNo central component, loose couplingFlow is implicit, hard to debug and changeTwo or three steps, stable flow
Workflow engine (Temporal, Step Functions)Durable state, retries, timers built inNew platform dependency, determinism rulesMany sagas across teams
2PCTrue atomicity and isolationBlocking, couples availabilityAll participants in one database system
Single service owns the dataPlain ACID transactionLarger service boundaryThe steps do not really need separate services

Pitfalls in production

Leadership lens

Check yourself

  1. Why are compensating transactions not the same as rollbacks?

    Show answer

    The original step committed and its effects may have been seen or acted on, so the compensation is a new forward action that semantically cancels it rather than erasing it.

  2. Why place a pivot step and only retriable steps after it?

    Show answer

    It guarantees the saga can either fully compensate before the point of no return or complete afterwards by retrying, so it never needs to undo something irreversible.

  3. What isolation problem do sagas have, and name one countermeasure?

    Show answer

    Other transactions can see or act on intermediate states; a semantic lock such as a PENDING status lets other operations recognise and avoid in-flight data.

  4. Why must every saga step be idempotent?

    Show answer

    Messages are delivered at least once and the orchestrator retries after timeouts, so the same command can arrive more than once.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →