Debugging with an Agent as Your Pair

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Agents are excellent debugging hands: they read logs, grep code, write reproduction scripts, run bisects and try experiments faster than you can type. They are unreliable debugging heads: they jump to the most common cause and "fix" symptoms.
Keep the hypothesis ledger yourself. Write down the symptom, the hypotheses, what each predicts and what evidence would rule it out, and use the agent to gather that evidence.
Ask for a failing test or a reproduction before any fix. A fix without a reproduction is a guess with a commit message.
Reject symptom fixes: added retries, broader try/except, longer timeouts, or deleted assertions are often the agent making the error go away, not making the bug go away.
Use parallelism on purpose: several sub-agents can test competing hypotheses at once, as long as each reports evidence rather than conclusions.

How it works

Hands, not head

Debugging is a loop of observing, hypothesising, predicting and testing. Agents speed up the mechanical parts dramatically: "find every place we call datetime.now() in the checkout package", "write a script that sends 500 concurrent requests and counts duplicate orders", "run git bisect between v2.3 and v2.4 using this test". They are weaker at the parts that need judgment: noticing which observation is surprising, keeping several explanations alive at once, and refusing a plausible fix that does not explain all the evidence. Models are trained on huge amounts of code and bug reports, so they reach for the statistically common cause; your bug may be the rare one.

The ledger

Keep a short written ledger, in a file the agent can read too:

Symptom, precisely: "about 0.3 percent of orders are created twice, only for mobile clients, since Tuesday".
Hypotheses, each with a prediction: "H1, client retries after timeout: duplicates share a client request ID and arrive 10 to 30 s apart." "H2, a queue redelivers: duplicates share a message ID and have no client retry in the logs." "H3, a race in the order handler: duplicates arrive within 50 ms."
Evidence gathered, and which hypotheses it eliminates.
Next experiment: the one that best separates the remaining hypotheses.

The agent gathers evidence; you update the ledger. This keeps the reasoning visible and stops the session from drifting into a fix for whichever hypothesis the agent liked first. Early wrong guesses stick even for experts: in Cloudflare's November 2025 outage the team first believed a huge DDoS attack was under way, because errors came and went as a database cluster was updated node by node and the status page happened to fail too (Cloudflare).

Reproduce first

Ask the agent to turn the symptom into a failing test or a reproduction script before it changes any code. Anthropic's own guidance for Claude Code says the same: give the agent a check it can run, and describe the symptom, the likely location and what fixed looks like (Claude Code best practices). If it cannot reproduce the bug, that is information: your hypotheses about the trigger are wrong or incomplete. If it can, you have an oracle for the fix and for bisecting, and a regression test for free.

Recognise symptom fixes

Some changes make failures disappear without fixing anything. Agents propose them often, because they make tests pass:

Adding a retry around the failing call (now the duplicate order is retried too).
Catching and ignoring the exception.
Increasing a timeout until the flaky test passes.
Loosening an assertion or marking a test as skipped.
Adding a sleep to avoid a race.

Each can be right in some situations, but each needs an explanation of the root cause first. The same guidance suggests asking explicitly to "address the root cause, don't suppress the error" (Claude Code best practices). Ask: "What is the root cause, what evidence shows it, and why does this change address the cause rather than the symptom?"

Parallel hypotheses

With sub-agents, you can test hypotheses side by side: one checks client logs for retries, one checks queue delivery counts, one writes a concurrency test. Ask each for evidence in a fixed form, such as counts, log lines and file locations, not "I believe the cause is". You merge the evidence into the ledger.

Down the stack

Layer 1The contract

Agent-assisted debugging promises the same outcome as any debugging: a root cause supported by evidence, a reproduction that fails before the fix and passes after, and a fix that addresses the cause. The agent's role is to gather evidence and run experiments; the human is accountable for the conclusion.

Layer 2The mechanism

Agents debug through tools: shell commands (grep, rg, jq over logs, git log -S to find when a string appeared, git bisect run with a test script), test runners, database read queries, and increasingly browser and runtime tools such as Chrome DevTools over MCP (public preview since September 2025), which let an agent read console errors, network requests and performance traces. git bisect run ./repro.sh is a good example of a perfect agent task: the script's exit code (0 good, 1 to 127 except 125 bad, 125 skip, anything above 127 aborts, per the git documentation) drives a binary search over commits, about log2(N) steps, so 1,000 commits take about 10 test runs.

Layer 3The machine

Much debugging evidence comes from the operating system, and agents can collect it quickly if allowed: strace -f -e trace=network shows every connect and sendto a process makes; ss -tanp lists sockets and their states, revealing connection-pool exhaustion as hundreds of sockets in CLOSE_WAIT; perf top shows which functions consume CPU; core dumps capture memory at a crash. The machine's view is ground truth in a way the agent's interpretation is not, so ask for raw output alongside any summary, and read the raw output yourself for the key experiments.

Why it's built this way

Debugging needs many cheap experiments, which agents run quickly.
Agents pattern-match to common causes and optimise for green tests.
So the reasoning (hypotheses, predictions, eliminations) must stay explicit and human-owned.
Reproductions turn hypotheses into checkable oracles for both the fix and the agent.
Raw evidence from the machine keeps everyone honest, human and agent alike.

Diagram

sequenceDiagram
  participant H as Human (ledger)
  participant A as Agent (hands)
  participant S as System
  H->>A: symptom and hypotheses with predictions
  A->>S: grep logs, count, write repro
  S-->>A: raw evidence
  A-->>H: evidence, not conclusions
  H->>H: eliminate hypotheses, pick next experiment
  H->>A: run experiment or bisect with repro
  A-->>H: failing test, then candidate fix
  H->>H: does the fix explain all the evidence?

Worked example

Duplicate orders: 0.3 percent of mobile orders since Tuesday, about 400 a day. The engineer opens a session and writes the ledger with three hypotheses: client retries (H1), queue redelivery (H2) and a race in the handler (H3).

First, the agent proposes a fix immediately: "add a unique constraint on (user_id, cart_id)". The engineer declines: that might stop the symptom, but it does not explain Tuesday, and it would also block legitimate re-orders from the same cart. Instead, they ask for evidence. The agent queries one day of duplicates: in 97 percent of pairs the client request IDs differ, which argues against simple client retries with the same ID (H1 as stated), and the gaps cluster between 9 and 11 seconds. Queue logs show each message delivered once (H2 unlikely). The 10-second gap does not fit a 50 ms race (H3 unlikely).

New hypothesis H4: something on the client creates a new request after about 10 seconds. The agent searches the mobile app's changes since Tuesday and finds a release that added a 10-second "still working?" timeout, after which the app re-submits the checkout with a new request ID. The agent writes a reproduction against staging using a slowed backend: two orders, 10 seconds apart. The real fix is in the client (reuse the request ID on re-submit) plus server-side idempotency keyed on a checkout session ID. The reproduction fails before and passes after. The quick unique-constraint fix would have hidden the problem and blocked real customers.

In the wild

Cloudflare, 2025-11: the wrong first hypothesis cost hours. A database permissions change made a query return duplicate rows, so a bot-management feature file doubled in size and passed a hard limit of 200 features (about 60 were in use). The proxy crashed. Because the file was regenerated every five minutes by a cluster being updated gradually, failures flickered, and the team first suspected a DDoS. The file was identified as the trigger about two hours after onset and total impact lasted about 5 hours 46 minutes (Cloudflare).
Replit, 2025-07: an agent with write access to production deleted a live database. Replit's AI coding tool reportedly deleted a production database despite an instruction to freeze code, and the vendor responded by working on automatic separation of development and production databases (The Register). Treat the founder's account as unverified, but the control it points to is real: investigation agents get read-only access.

Trade-offs

Approach	Strengths	Weaknesses	Choose when
Agent fixes directly	Fastest when the cause is common	Symptom fixes, wrong root cause	Trivial, well-understood errors
Human ledger, agent gathers evidence	Fast and rigorous	Requires discipline	Most real bugs
Parallel sub-agents per hypothesis	Tests many ideas at once	More cost, merging effort	Several plausible causes, urgent bugs
Human only	Full understanding	Slow on mechanical work	Sensitive systems where agents lack access

Pitfalls in production

Accepting the first plausible fix. Fix: require the root cause and evidence that rules out the alternatives.
No reproduction, so nobody can tell whether the fix works. Fix: failing test or script before changing code.
Retries, catches, sleeps and timeouts used as fixes. Fix: treat them as suspicious until the cause is explained.
Summaries instead of evidence. Fix: ask for raw counts, log lines and command output.
Giving a debugging agent write access to production to "check something". Fix: read-only access and replicas; changes go through normal deploys. In July 2025 a Replit agent reportedly deleted a live database despite a code freeze (The Register).

Leadership lens

Teach the ledger habit: symptom, hypotheses with predictions, evidence, next experiment. It works with or without agents.
Make "reproduction first" a team norm for bug fixes, enforced in review: every bug-fix pull request includes a test that failed before.
Give agents good read-only debugging tools (log queries, trace search, replicas), and keep production writes out of reach.
In postmortems, note when an early symptom fix delayed finding the root cause; it is a common, teachable pattern.

Check yourself

An agent proposes wrapping a failing HTTP call in three retries. What do you ask before accepting it?
Show answer
What is the root cause of the failures, what evidence supports it, and is the call idempotent? Retries may hide the cause and can duplicate side effects.
Why should the agent report evidence rather than conclusions?
Show answer
Conclusions hide the reasoning and the agent's bias toward common causes; raw evidence lets you check it and eliminate hypotheses yourself.
Why is a reproduction so valuable when working with an agent?
Show answer
It is an objective oracle: it proves the bug exists, drives bisection, shows whether a fix works and becomes a regression test.
How does git bisect run decide which half to keep?
Show answer
It runs the script at the midpoint commit and uses the exit code: 0 marks it good, 1 to 127 except 125 marks it bad, and 125 skips it.

Sources and further study

Primary

Cloudflare, Outage on November 18, 2025: a full timeline showing how a plausible but wrong hypothesis (DDoS) slowed diagnosis.
DORA, 2024 Accelerate State of DevOps report: finds AI adoption raises individual productivity but hurts delivery stability, and that small batches and testing still matter.

Systems docs

Claude Code best practices: verification checks, plan mode and fixing root causes rather than suppressing errors.
Chrome DevTools MCP announcement: which browser evidence an agent can read (console, network, traces).

Books

Site Reliability Engineering, Google, chapter 12 "Effective Troubleshooting": theory, hypothesis testing and why negative results are useful.
Observability Engineering, Majors, Fong-Jones and Miranda, chapter 8 "Analyzing Events to Achieve Observability": debugging from first principles.
Building Secure and Reliable Systems, Google, chapter 15 "Investigating Systems": useful logs and robust, secure debugging access.

Free resources

A debugging manifesto (Julia Evans): habits for debugging well, all of which apply with agents.
The Debugging Book (Andreas Zeller): systematic, scientific debugging with runnable examples.
git-bisect documentation: automated bisection with git bisect run.
Chrome DevTools MCP (Chrome for Developers): giving agents the browser's console, network and performance data.
Best practices for Claude Code (Anthropic): explore, plan, test-first workflows with an agent.

Open in the interactive app: practice, quiz, mock interview →