Engineering Atlas › Low-Level Design

Code a workflow engine

15 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional:

Non-functional:

Out of scope: parallel branches, timers and signals, sub-workflows, versioning of running definitions, and a real database. Mention them in Extensibility.

Entities and relationships

The status field is the state machine. RUNNING moves to COMPLETED when the last step succeeds, or to COMPENSATING when a step fails for good. COMPENSATING moves to ROLLED_BACK when the done stack is empty, or to COMPENSATION_FAILED when an undo exhausts its attempts. Three states are terminal, and nothing leaves them. Within RUNNING, a failed attempt does not change status; it moves next_at into the future.

The engine is a composition: it owns its steps, and it uses the store, clock and rng through narrow interfaces, so each can be replaced by a fake.

Class diagram

classDiagram
    class Engine {
      -steps: Step[]
      +start(id)
      +run(id) Status
      +status(id) Status
      +history(id) string[]
      +nextRunAt(id) int
    }
    class Step {
      +name: string
      +action(key)
      +compensate(key)
      +maxAttempts: int
      +baseDelay: int
      +factor: float
      +maxDelay: int
      +timeout: int
    }
    class RunRecord {
      +status: Status
      +index: int
      +attempt: int
      +nextAt: int
      +done: int[]
      +history: string[]
    }
    class Store {
      <<interface>>
      +load(id) RunRecord
      +save(id, record)
    }
    class Clock {
      <<interface>>
      +now() int
    }
    Engine "1" *-- "many" Step
    Engine --> Store
    Engine --> Clock
    Store "1" o-- "many" RunRecord
    MemoryStore ..|> Store
    FakeClock ..|> Clock

Key flows

One call to run loops: load the record, stop if it is terminal or not yet due, otherwise perform a single attempt, save, and repeat. The sequence below shows a step that fails once, then succeeds after its backoff, followed by a later step that fails permanently and triggers compensation.

sequenceDiagram
    participant C as Caller
    participant E as Engine
    participant S as Store
    participant A as Step action
    C->>E: run(w)
    E->>S: load(w)
    E->>A: action(key)
    A-->>E: error
    E->>S: save next_at = now + delay
    E-->>C: RUNNING
    C->>E: run(w) after the delay
    E->>S: load(w)
    E->>A: action(key)
    A-->>E: ok
    E->>S: save index + 1
    Note over E,A: next step fails with a permanent error
    E->>S: save status COMPENSATING
    E->>A: compensate previous step
    A-->>E: ok
    E->>S: save ROLLED_BACK
    E-->>C: ROLLED_BACK

Note that run never sleeps. A waiting run just returns RUNNING with a next_at, and something else (a scheduler, a timer wheel, a test) calls it again. That separation is the reason the engine is testable.

Design patterns used and why

Concurrency and edge cases

In the wild

Down the stack

Layer 1The contract

run(id) returns the status after performing every attempt due at the current clock time. It never performs an attempt before next_at. For a step with maxAttempts = n, the action is called at most n times in one forward pass, at least once per resume in the worst case, and never after the step has succeeded and been saved. If a step fails for good, undo actions run in reverse order of completion, each at most maxAttempts times, and the failed step itself is not undone. A terminal status is final. Each attempt appends exactly one name:outcome:n entry to the history, so the audit trail matches the calls made.

Layer 2The mechanism

The delay after the nth failure is min(max_delay, base * factor^(n-1)), multiplied by a random fraction when jitter is on. AWS's analysis of contention found that exponential backoff alone still clusters retries and that adding jitter spreads them and reduces total work (AWS Architecture Blog). Real engines expose the same four knobs. Temporal defaults to a 1 second initial interval, a coefficient of 2.0, a cap of 100 times the initial interval, and unlimited attempts (Temporal docs). AWS Step Functions defaults to IntervalSeconds 1, MaxAttempts 3 and BackoffRate 2.0, and its JitterStrategy defaults to NONE (Step Functions docs). A timeout in Step Functions is itself an error name, States.Timeout, which a Retry block can match.

Where this lesson stores a record and rewrites it, Temporal and Cadence store an append-only event history and rebuild state by replaying workflow code against it, which forces the code to be deterministic. Temporal caps one execution at 51,200 events and offers Continue-As-New to start a fresh history (Temporal docs). The record design is simpler; the log gives audit and rewind. The saga idea itself is older: Garcia-Molina and Salem proposed splitting a long transaction into sub-transactions, each with a compensating one, in 1987 (Sagas, SIGMOD 1987).

Layer 3The machine

A save is the expensive operation. On Postgres a commit waits for the write-ahead log fsync, roughly 0.1 to 1 ms on NVMe and several milliseconds on network storage, so a five-step workflow with no failures costs five commits, a few milliseconds of engine overhead beside steps that make network calls of 10 to 500 ms. Saving after every attempt is therefore cheap relative to the work and buys exactness at restart. A record of this shape is a few hundred bytes of JSON, so 10 million active runs is a few gigabytes. The scheduling side is the costlier part: finding runs whose next_at has passed needs an index on (status, next_at) and a poller, or a delay queue, so due runs are found in O(log n) instead of a table scan. The poller must tolerate the same row being claimed twice, which returns to compare-and-set.

Why it's built this way

  1. Business processes touch several systems that cannot share one transaction, so the engine owns the "what happened so far" record and compensation replaces rollback.
  2. Processes and machines crash mid-step, so state is saved after every attempt to a store outside the process.
  3. A crash between an effect and its save cannot be eliminated, so delivery is at-least-once and steps carry idempotency keys.
  4. Failures are often correlated overload, so retries use capped exponential delay with jitter, and permanent errors skip retries.
  5. Waiting must not tie up a thread, so the engine records next_at and returns, and a clock and rng injected from outside keep all of it testable.

Extensibility

Leadership lens

Check yourself

  1. A worker crashes after the payment step charged the card but before it saved progress. What happens on restart, and what prevents a double charge?

    Show answer

    The record still says the payment step is pending, so the engine runs it again (at-least-once). The action sends a stable idempotency key derived from the workflow id and step name, and the payment service returns the first result for a repeated key.

  2. Why does the engine record next_at and return instead of sleeping for the backoff?

    Show answer

    Sleeping holds a thread and is lost on crash. A saved next_at survives restarts, frees the worker, and lets tests drive time with a fake clock.

  3. Why add jitter to exponential backoff, and what does it cost?

    Show answer

    Without it, many failed runs retry at the same instants and re-create the overload. Jitter spreads them out. The cost is a less predictable individual delay and harder exact-timing tests, solved by an injected rng.

  4. Step 4 of 5 fails permanently. Which compensations run, in which order, and is step 4's undo among them?

    Show answer

    Steps 3, 2, 1 in that order, skipping any step without a compensation. Step 4 never completed, so it is not in the done stack; if it can leave partial effects, its own action must clean up or tolerate retries.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →