Engineering Atlas › Problem Solving & Incidents (AI era)

Naming the Problem Shape

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Shapes leave signatures

Problems are easier to name from evidence than from descriptions, because descriptions carry the domain's nouns and the reporter's guess. Evidence carries the shape.

The five questions

When you meet a new feature or bug, ask these in order:

  1. What is scarce? Seats, drivers, stock, money, a lock, a GPU. If anything is scarce, you have contention.
  2. What can repeat? Any network call can be retried, any message redelivered, any button pressed twice (retried steps that are not idempotent leave wrong counts and duplicate messages, per the Azure Architecture Center). If repeating it is harmful, you have duplicates.
  3. What must happen in order? If later events must not be overwritten by earlier ones, you have ordering.
  4. Who must hear about it, and how many? One receiver is a call; thousands is fan-out.
  5. What can fail halfway? If an operation spans two systems, you have partial failure and multi-step consistency.

Then two scale questions: is any key much hotter than the rest, and can total demand exceed capacity?

Stacked shapes

Real problems are rarely one shape. A webhook from a payment provider is duplicates (providers retry), ordering (events can arrive out of order) and partial failure (your handler may crash after updating the database but before replying). Stripe documents the first two outright: the same event can arrive more than once, and delivery order is not guaranteed (Stripe webhooks). It also retries live-mode deliveries for up to three days, so a slow or failing handler sees duplicates long after the original. Naming only one leads to a fix that holds for a week. Write all of them down and rank them by damage: a duplicate refund costs money, an out-of-order status costs a support ticket.

Shapes as prompts

In the AI era, a named shape is a compact, precise prompt. "Make the webhook handler idempotent on event.id, re-fetch the object from the provider's API instead of trusting arrival order, and write the outbox row in the same transaction" gets a correct implementation from a coding agent in one pass. Note the changed ordering rule: Stripe's created field has one-second resolution, so distinct events can share a timestamp, and its docs say not to use it for ordering or de-duplication; track event IDs and fetch current state instead (Stripe webhooks). A shape names the problem, but the mechanism still has to be checked against the provider's real guarantees. "Fix the webhook bugs" gets a plausible patch for whichever symptom the agent saw first.

Down the stack

Layer 1The contract

The practice promises a written problem statement that names every shape present, with the evidence for each and a ranking by impact. A reviewer can check it: every shape must point to a symptom or a log query, and every shape must have a mechanism in the design.

Layer 2The mechanism

Shape detection is mostly grouping and comparing. Duplicates: group applied events by idempotency key and count; a service that implements keys properly stores the first response and rejects the same key with different parameters, which gives you a second signature to look for (Stripe error handling). Ordering: for each key, walk events in arrival order and flag any sequence number lower than the highest already seen (a log such as Kafka's only orders within one partition, so "arrival order" means per partition; Kafka docs). Contention: for each key, sort acquire and release intervals by start time and flag overlaps between different actors. Hot spots: count events per key and compare each key's share to the mean, for example flag any key above 5 times the average or above a fixed share such as 20 percent. These are the same algorithms a database uses: unique indexes find duplicates, version columns find ordering violations, lock managers find overlaps.

Layer 3The machine

The evidence lives in logs, traces and tables, and the cost of finding it depends on how it is stored. Grouping 50 million log lines by request ID in a columnar store reads only the ID column, a small fraction of the data (the exact size depends on the ID type and compression), so it can finish in seconds. The same query on raw JSON logs in object storage reads every byte of every line. That is why structured logs with a stable request ID, entity key and sequence number on every line pay off: they turn shape detection from an archaeology project into a one-line query.

Why it's built this way

  1. Descriptions of problems carry domain words and guesses, which hide the shape.
  2. Data carries the shape directly, because shapes are about keys, actors, order and counts.
  3. So the reliable way to name a shape is to look at grouped, ordered evidence.
  4. Grouping and ordering need stable identifiers on every event.
  5. Hence the habit: log request IDs, entity keys and sequence numbers everywhere, and ask the five questions before designing.

Diagram

flowchart TD
  E["Events: key, actor, request id, seq"] --> G1["Group by request id"]
  E --> G2["Per key, arrival order"]
  E --> G3["Per key, intervals"]
  E --> G4["Count per key"]
  G1 --> D["Duplicates: id applied twice"]
  G2 --> O["Ordering: seq went backwards"]
  G3 --> C["Contention: overlapping holders"]
  G4 --> H["Hot spot: share far above mean"]

Worked example

A food delivery company sees "random" issues: some couriers get two orders at the same restaurant at once, some customers see "delivered" flip back to "on the way", and the Friday-night dispatcher runs slowly. Each team has its own theory.

An engineer exports one evening's dispatch events, 1.2 million rows with courier_id, order_id, request_id, seq and timestamps, and runs four grouped queries. 0.3 percent of assignment request IDs appear twice: duplicates from mobile retries. 0.8 percent of orders have a status with a lower seq applied after a higher one: ordering, from status updates sent through two different queues. Overlapping assignments on the same courier: contention, 140 cases, all within 50 ms, consistent with a check-then-write race. And one key, the "city:central" geo cell, takes 31 percent of dispatch lookups: a hot spot that explains the slow Friday nights.

Four shapes, four known fixes: idempotency keys on assignment, a per-order version check, compare-and-set on courier status, and splitting the hot cell into smaller cells. Ranking by impact puts contention first (two orders on one bike means late food), then ordering, then the hot spot, then duplicates.

In the wild

Trade-offs

MethodStrengthsWeaknessesChoose when
Five questions on the designCheap, catches shapes before code existsDepends on imaginationNew features and design reviews
Evidence queries on logsObjective, finds shapes you did not imagineNeeds structured logs with IDsBugs and incidents already happening
Invariant checks in productionContinuous detectionMust know the invariant in advanceMoney, inventory, anything audited
Asking an agent to classifyFast first pass over long reportsConfident and sometimes wrongTriage, always followed by evidence

Pitfalls in production

Leadership lens

Check yourself

  1. Customers report that their profile photo sometimes reverts to an older one a minute after uploading a new one. Which shape, and what evidence would confirm it?

    Show answer

    Ordering. Confirm by finding, for one user, an older upload's write applied after a newer one, for example via timestamps or version numbers in the storage logs.

  2. Why do stacked shapes matter for how you rank work?

    Show answer

    Each shape has a different impact and fix; ranking by damage lets you fix the costly one first instead of whichever was noticed first.

  3. What is the difference between a hot spot and overload, and why does it matter?

    Show answer

    Overload is total demand above total capacity; a hot spot is one key above its share while others idle. Adding machines fixes overload but not a hot spot, which needs repartitioning or caching.

  4. Why is "make the handler idempotent on event.id" a better prompt for an agent than "fix the duplicate bug"?

    Show answer

    It names the shape and the mechanism, so the agent implements a known, testable solution instead of patching the one symptom it can see.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →