Engineering Atlas › Problem Solving & Incidents (AI era)

Reproducing the Bug

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Capture the conditions

Before touching code, collect everything that made the failing run different from the thousands that passed. That means the exact request or input file, the build and dependency versions, configuration and feature-flag state at the time, the data the request touched, the host and its resource limits, and the time. Production systems that log a request ID next to the build SHA and flag snapshot make this a query instead of an archaeology project. If the process crashed, a core dump is the single most valuable artefact: it holds every thread's stack and the heap at the moment of death.

Make it happen at all

Rare bugs need volume. Run the suspect path in a loop thousands of times, raise concurrency above production levels, shrink timeouts, and inject latency or failures at the boundaries you suspect. A failure rate of 1 in 5,000 becomes one every few seconds in a tight loop. The goal at this stage is any failure, however ugly the harness.

Make it happen every time

Then remove nondeterminism one source at a time. Inject the clock instead of reading wall time. Fix random seeds. Pin hash seeds (PYTHONHASHSEED) and remember that Go deliberately randomises map iteration order. Sort directory listings. Reduce CPU count or GOMAXPROCS to change scheduling pressure. Disable ASLR with setarch -R if the bug depends on addresses. Each source you pin is also a clue: if pinning the clock makes the bug vanish, time is part of the mechanism.

Make it small

A 48,000-row CSV that crashes the importer is a reproduction, but nobody can reason about it. Shrink the input until every remaining part is necessary. Do it by hand by halving, or let a tool do it: delta debugging (ddmin) tries chunks and their complements, keeps any smaller input that still fails, and refines the granularity when nothing smaller fails. Property-based testing frameworks such as Hypothesis do the same automatically when they "shrink" a counterexample.

Turn it into a test

The final reproduction belongs in the test suite. It should fail on the broken build, pass on the fixed build, and run in seconds. If it needs the full stack, mark it as an integration test but keep it.

Down the stack

Layer 1The contract

A reproduction is a script plus an environment description that, when run, shows the failure with a stated probability, ideally 1, and stops showing it when the fix is applied. Testable: someone else can run it on a clean machine, the failure signature matches production (same exception, same corrupted field), and the reproduction fails on the commit before the fix and passes on the commit with it. A repro with an unknown failure rate cannot prove a fix; if it fails 1 time in 20, you need roughly 60 clean runs before "fixed" is believable at 95% confidence.

Layer 2The mechanism

Delta debugging, from Andreas Zeller's work, needs only an oracle fails(input). Split the input into n chunks. If one chunk alone fails, recurse on it. Otherwise, if removing one chunk still fails, drop that chunk. Otherwise double n and retry, up to single elements. The result is 1-minimal: removing any single element makes it pass. It does not guarantee the globally smallest input, and the worst case needs n squared plus 3n oracle calls for n elements (Zeller and Hildebrandt, 2002), but on real inputs it is usually far cheaper. The oracle should return three outcomes, not two: pass, fail, and unresolved for a candidate that fails differently or cannot run. The Debugging Book's implementation counts a run as a failure only if it raises the same exception type with the same message; a different exception is reported as unresolved, and a run that raises nothing is a pass, so in neither case is the reduction step accepted (Debugging Book).

When the trigger is scheduling, not data, record and replay helps. rr, from Mozilla, records a Linux process tree by capturing every source of nondeterminism: syscall results, signal delivery points and the exact point of each context switch. It runs all threads on one core at a time and uses the CPU's retired-conditional-branch performance counter, paired with the register state, to find the same instruction again on replay: it is the only Intel counter the authors found to be deterministic (Pillai et al., rr paper). A recorded failure replays identically under gdb, including reverse execution. The paper reports slowdowns under 2x on its important low-parallelism workloads, while workloads that are consistently highly parallel slow down a lot more (Pillai et al., rr paper). It needs hardware performance counters: the project lists Intel Nehalem or later, some AMD Zen parts and some AArch64 cores, and virtual machines only if they virtualise the counters (rr README).

For crashes, the core dump is an ELF file. PT_LOAD segments hold the process's memory regions; PT_NOTE segments carry NT_PRSTATUS entries (one per thread, with registers), NT_FILE (which files were mapped where) and more. gdb binary core reconstructs every thread's stack from those registers, as long as you kept the exact binary and its debug symbols.

Layer 3The machine

Nondeterminism enters at a small number of machine-level doors. clock_gettime usually does not even enter the kernel: it reads the vDSO, a page the kernel maps into every process. getrandom and /dev/urandom supply entropy. The rdtsc instruction reads the cycle counter directly. ASLR randomises the base of the stack, heap and shared libraries on every execve. The scheduler picks which runnable thread gets a CPU at each tick or wake-up, and on a 16-core host two threads may interleave in ways a 2-core laptop never shows. Readdir order depends on the filesystem's on-disk layout, not on file names.

Core dumps depend on kernel settings: ulimit -c must allow them, /proc/sys/kernel/core_pattern decides whether the dump goes to a file or is piped to a handler like systemd-coredump when it starts with |, and /proc/<pid>/coredump_filter selects which mappings are written. RLIMIT_CORE is not enforced for piped dumps, so the handler has to enforce its own size cap (core(5)). A process with a 20 GB heap produces a 20 GB core, which matters in containers with small disks.

Why it's built this way

  1. You cannot verify a fix without a failing case, so reproduction comes before repair.
  2. Production carries hidden inputs (time, entropy, scheduling, config), so you capture or pin them until the failure is a function of what you control.
  3. Large reproductions are slow and hard to reason about, so you minimise automatically with an oracle.
  4. Some nondeterminism cannot be pinned from user space, so tools like rr record it at the kernel boundary and replay it exactly.

Diagram

flowchart TD
    P["Production failure"] --> C["Capture: input, versions, flags, core dump"]
    C --> A["Make it happen at all: loop, stress, inject"]
    A --> D["Make it deterministic: pin clock, seeds, scheduling"]
    D --> M["Minimise with ddmin oracle"]
    M --> T["Regression test fails before fix"]
    D -->|"cannot pin"| R["Record with rr, replay in gdb"]
    R --> M

Worked example

An import job crashes on about 1 in 5,000 uploaded files. The logs give a request ID; the request ID leads to the uploaded CSV in object storage and the build SHA. Run locally against that SHA, the file crashes the importer every time, so the bug is deterministic in the data, not timing.

The file has 48,000 rows and one run takes 0.3 seconds. The oracle is "the importer raises IndexError in parse_row" (matching the signature matters: a different exception from a malformed half-file does not count). ddmin first tries halves, keeps whichever half still fails, and continues. Around 17 halvings would isolate a single culprit row; this bug needs two rows, a header-like row with a byte-order mark in the middle of the file and a row after it with an empty last column. Interacting causes cost extra rounds of complements, and the run ends at about 150 oracle calls, roughly 45 seconds. A human halving by hand would likely have stopped at "rows 30,000 to 31,000" and missed that two separate rows were required.

The two-row file becomes a unit test, and its shape explains the mechanism: the BOM resets the column count used for the next row.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
Replay captured inputFast, exact, cheapNeeds the input to have been capturedData-dependent bugs
Stress loop with injectionFinds rare timing bugsSlow, may change the bugRaces, timeouts, retries
Record and replay (rr)Exact replay, reverse debuggingLinux only, needs hardware performance counters (Intel, some AMD and AArch64 cores), one thread at a time during recordingNative code, rare crashes
Deterministic simulationExplores schedules and faults reproduciblyLarge up-front investment in architectureDistributed systems you own end to end
Core dump analysisNo reproduction needed for the crash stateShows the end, not the pathCrashes you cannot trigger again

Pitfalls in production

Leadership lens

Check yourself

  1. Your repro fails 1 time in 10. After the fix it passes 10 runs in a row. Is it fixed?

    Show answer

    Not convincingly. A 1-in-10 bug passes 10 runs about 35% of the time by chance; you need roughly 30 or more clean runs for 95% confidence.

  2. Why does ddmin also try complements, not just chunks?

    Show answer

    When the failure needs elements from two different chunks, no single chunk fails, but removing an unrelated chunk still fails, so complements make progress.

  3. Why can rr replay a race faithfully even though it runs threads on one core?

    Show answer

    It records exactly where each context switch happened, measured in retired branches, and reproduces that same interleaving on replay.

  4. A crash happened once and never again. What do you need to have set up in advance to debug it?

    Show answer

    Core dumps enabled with a working core_pattern, the exact binary, and its debug symbols archived by build ID.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →