The Paradox: Every System Has the Same Problems

In one minute

• Every team believes its domain is special, and it is: the vocabulary, the regulations and the cost of a mistake differ. Underneath, the same few problems repeat: contention, duplicates, ordering, fan-out, hot spots, overload, partial failure, multi-step consistency, finding things fast, and trust.
• The paradox: the more domains you see, the fewer distinct problems there seem to be. A seat at a concert, a driver at an airport and the last pair of shoes in stock are one problem with three costumes.
• Naming the shape is the move that turns a scary new problem into a familiar one, with decades of known solutions, failure modes and trade-offs attached.
• The domain still matters, but it decides the parameters (how long a hold lasts, what a duplicate costs, who gets told), not the mechanism.
• In the AI era this matters more: an agent can write the mechanism in minutes once you name it correctly, and it will confidently write the wrong one if you don't.

How it works

Why domains look different

A payments engineer talks about authorisations, captures and chargebacks. A ride-hailing engineer talks about dispatch, ETAs and surge. A chat engineer talks about presence, read receipts and fan-out. Each group has its own conferences, its own jargon and its own war stories, so it is natural to assume each needs its own way of thinking.

Strip the nouns off the war stories and they rhyme. "A customer was charged twice" and "a rider got two cars" and "the group got the message twice" are the same sentence: a request was retried, and the side effect was not idempotent. Payment providers say the same thing in their own docs: a connection error leaves you not knowing whether the request ran, so the API accepts a client-chosen key to make a retry safe (Stripe API docs), and Stripe's webhook guide tells receivers to expect the same event more than once (Stripe webhooks). "We sold 103 tickets for 100 seats" and "two riders were matched to one driver" are the same sentence: two writers raced on one scarce thing and the check and the write were not atomic.

The ten shapes

This course uses ten shapes, listed in curriculum/problem-shapes.md and on the home page:

1. Contention: many writers want one thing.
2. Duplicates: the same request arrives more than once.
3. Ordering: events arrive in a different order than they happened.
4. Fan-out: one event must reach many receivers.
5. Hot spots: skew, where one key gets far more than its share.
6. Overload: demand exceeds capacity.
7. Partial failure: you cannot tell whether something happened.
8. Multi-step consistency: several changes must all happen or none.
9. Finding things fast: search, lookup and nearest-neighbour.
10. Trust but verify: checking that what happened is what should have happened.

The list is not sacred. What matters is having a short list in your head and reaching for it every time.

What the domain decides

Once the shape is named, the domain supplies the numbers and the policy. For contention, the mechanism is a hold with an expiry plus an atomic check-and-set. The domain decides how long the hold lasts (a concert seat: ten minutes while you pay; a driver: fifteen seconds while they accept; a stock unit: thirty minutes in a cart), what happens when the hold expires, and whether overbooking is allowed on purpose (airlines do it, cinemas do not). Mistaking a parameter difference for a mechanism difference is how teams end up building three reservation systems that each have a different bug.

Why this is the skill of the AI era

Coding agents are very good at producing a mechanism once it is named: "add an idempotency key with a unique constraint and return the stored response on conflict" is a ten-minute task. They are much worse at noticing that a feature request about "group gift cards" is secretly a multi-step consistency problem. The engineer who names the shape writes the right spec, asks the agent for the right mechanism, and knows which tests prove it. The engineer who doesn't gets fast, plausible code for the wrong problem.

Down the stack

Diagram

flowchart LR
  D["New problem in your domain"] --> S["Strip the nouns"]
  S --> N["Name the shape"]
  N --> M["Known mechanisms"]
  N --> F["Known failure modes"]
  M --> P["Set domain parameters"]
  F --> T["Tests for each failure"]
  P --> B["Design or agent spec"]
  T --> B

Worked example

A product manager asks for "group gift cards": several friends each chip in, and the card is issued when the target is reached. It sounds new. Strip the nouns: many payers each make a payment (several steps), the card must be issued exactly once (duplicates), contributions can race at the threshold (contention), and if one contribution fails after others succeed, someone must refund (multi-step consistency).

So the design is three known mechanisms. Each contribution carries an idempotency key, so a payer who taps twice pays once. The running total lives in one row updated with compare-and-set, so the contribution that crosses the target is the only one that triggers issuance. Issuance and refunds run as a saga with compensations, driven by an outbox so that the "target reached" event is never lost. Compensations can fail and be retried, so each one must itself be idempotent, and some steps (an email already sent) have no undo and should come last (Azure: compensating transaction pattern).

With 50,000 gift cards a day and five contributors each, that is 250,000 payments a day, about 3 a second on average and maybe 30 at peak: trivial load. The difficulty was never scale; it was naming the shapes. A team that skipped that step would likely ship a check-then-insert race and a missing refund path, then rediscover both in production.

In the wild

• Twilio, 2013: duplicate recharges from a read-only billing store. After a network blip, a Redis master used for balances came back read-only with balances at zero. Auto-recharge logic then kept trying to top up accounts it could not update, so 1.4% of customers were charged repeatedly. Twilio refunded them with a 10% credit and added fail-safes that block charges when balances are missing or unwritable (Twilio post-mortem). A payments duplicate, in a telecom costume.
• Knight Capital, 2012: one server missing a deploy. New code reached 7 of 8 servers; the eighth ran retired order-routing logic that kept sending orders, producing over 4 million executions in 154 stocks in about 45 minutes and a loss near $460 million. The SEC found no second-person review of the deploy (SEC order). The same family: an operation that repeats without the sender noticing, plus no check that the rollout matched the intent.
• Nasdaq, 2012: the Facebook IPO cross. Orders entered into the opening cross hit 496,000 against a tested 40,000; a late cancel arrived during a 20 ms calculation and forced a recalculation that kept looping. This is overload meeting ordering, in an exchange instead of an app (SEC order).

Trade-offs

Pitfalls in production

• Treating a parameter difference as a new problem, leading to three in-house reservation systems with three bugs. Fix: one shared component with domain parameters.
• Naming one shape and missing a second. Gift cards are duplicates and contention and multi-step consistency. Fix: run the whole list of ten for every new feature.
• Over-generalising: building a "universal workflow engine" because everything looks like a saga. Fix: share mechanisms, not giant frameworks; keep domain code in the domain.
• Assuming the agent knows the shape. Fix: put the shape and its failure modes in the spec, and ask for the tests first.
• Losing the domain's real constraints, such as legal rules on refunds, in the abstraction. Fix: parameters and policies stay explicit and reviewed by domain experts.

Leadership lens

• In design review, ask "which of the ten shapes is this?" before "which database?". It turns opinions into a checklist.
• Keep a team page that maps your domain's nouns to shapes: "booking = contention, webhook = duplicates + ordering". New hires and agents learn faster from it.
• Hire and promote for transfer: people who solved contention in logistics will solve it in fintech.
• When an incident review says "unique edge case", check whether it was a known shape with a missing mechanism. It usually was.

Check yourself

1. A rider app occasionally shows a trip as "cancelled" after it was "completed". Which shape is this, and what is the usual fix?

   Show answer

   Ordering: a late "cancelled" event overwrote a newer state. Attach a version or sequence number per trip and ignore events older than the stored version.

2. Why is "how long should a hold last?" a domain question, but "how do we stop two people holding the same seat?" is not?

   Show answer

   The atomic check-and-set mechanism is the same everywhere; the hold duration depends on how long the human or system on the other side needs, which only the domain knows.

3. A teammate says, "Our problem is unique, nobody has a real-time leaderboard for this game". How do you respond?

   Show answer

   Name the shapes: fan-out of score updates, hot spots on top players, finding things fast (top-k). Each has known solutions (sorted sets, sharded counters, periodic merge), so start there and focus effort on what is truly unique.

4. Why do these shapes also appear inside a CPU and in TCP?

   Show answer

   Because they come from universal constraints: delay, independent failure and finite capacity. Any system with those constraints, at any scale, meets the same problems.

Sources and further study

Primary

• Pat Helland, "Life beyond Distributed Transactions: An Apostate's Opinion" (CIDR 2007; ACM Queue version): entities as the unit of atomicity, and messaging between them.

Systems docs

• Stripe API: idempotent requests and Stripe webhooks.
• Apache Kafka design documentation: per-partition ordering and the idempotent producer.
• Azure: transient fault handling and compensating transaction.

Books

• Designing Data-Intensive Applications (1st ed.), Ch 8, "The Trouble with Distributed Systems", and Ch 9, "Consistency and Consensus" (ordering guarantees), by Martin Kleppmann.
• Release It! (2nd ed.), Ch 4, "Stability Antipatterns", and Ch 5, "Stability Patterns", by Michael Nygard.
• Design Patterns (1st ed.), Ch 1, "Introduction", by Gamma, Helm, Johnson and Vlissides: the idea of naming recurring solutions.

Other curricula

• System Design Primer: marks which HTTP methods are idempotent and notes that a queue such as SQS can deliver a message twice.

Watch

• You and Your Research (Richard Hamming): the 1995 talk on choosing important problems and carrying ideas between fields.

Free resources

• Making retries safe with idempotent APIs (AWS Builders' Library): the duplicates shape, solved the same way across many AWS services; a retry with the same client token returns an equivalent response, and a changed payload under that token is rejected.
• Designing robust and predictable APIs with idempotency (Stripe): how a payments company applies the same idea.
• Fallacies of distributed computing (Wikipedia): the universal constraints that produce the universal problems.
• Life beyond Distributed Transactions (Pat Helland): entities, activities and idempotence as domain-independent tools.
• How to Solve It (Wikipedia): Pólya's heuristic "have you seen this problem before in another form?".