Engineering Atlas › Problem Solving & Incidents (AI era)

Why Problem Solving Is the Job Now

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The shift in cost

Typing the code used to be a large share of the effort. An agent such as Claude Code, Cursor or Copilot's agent mode can now draft a feature, a migration or a test suite in minutes. What remains is the work around the code: understanding a vague request, choosing between designs, checking the result in every case that matters, and finding out why it fails in production.

That is problem solving. It was always the hard part; it used to hide inside "writing code", and now it is most of what is left.

Four loops

Think of any piece of engineering work as four loops that feed each other.

  1. Frame. Turn a symptom or request into a problem statement with a measurable success condition. "Checkout is slow" becomes "p99 of POST /checkout rose from 400 ms to 2.1 s for EU users since Tuesday's deploy; target is under 600 ms."
  2. Generate. Produce candidate solutions. An agent can produce three in the time a person writes one.
  3. Verify. Decide whether a candidate is correct, safe and maintainable. This needs an oracle: tests, invariants, a reference implementation, a metric, or a human who knows the domain.
  4. Diagnose. When verification or production says "wrong", find out why, and feed that back into framing.

Agents compress loop 2. Loops 1, 3 and 4 depend on context the agent often lacks: business intent, production behaviour, past incidents, unwritten rules. An agent will confidently solve the problem it was given, even the wrong one.

Where the time goes

Timestamp phase changes on real work, the way an incident timeline does, and look at the totals. You will often find generation is a small fraction of elapsed time, and that loops from verify back to generate ("not quite right, try again") are where the hours go.

One data point, with caveats: a 2025 randomised study by METR gave 16 experienced open-source developers 246 real issues in their own large repositories. With early-2025 AI tools allowed they took 19% longer, although they had forecast a 24% speedup and afterwards believed they had been about 20% faster (METR, July 2025). METR later marked those results out of date. Its February 2026 update, with 57 developers and over 800 tasks, estimated smaller slowdowns or modest speedups, but METR calls the data only very weak evidence, because many developers refused to work without AI and withheld tasks they did not want to do without it (METR, February 2026). The lesson is not "AI makes you slower" and not "AI makes you faster"; it is that perceived and measured speed can diverge, and that measuring productivity is hard, so keep your own phase data.

Checklist for any agent-assisted task

Down the stack

Layer 1The contract

The human process promises a decision with known confidence, not just a diff, stated so it is testable: "this change makes X true for inputs Y, does not change Z, and can be reverted within N minutes." If you cannot state the contract, neither you nor an agent can verify it. The Tech Lead makes sure each piece of work has one, and that someone is accountable for checking it. The same shape appears in Google's troubleshooting model: a problem report that states expected behaviour, actual behaviour and how to reproduce it (Google SRE book, ch. 12).

Layer 2The mechanism

An agent is a loop: a language model receives a prompt with the task, tool definitions and context, emits a tool call (read file, run command, edit file), the harness executes it, appends the result to the context, and repeats until the model emits a final answer. Correctness comes only from what the harness feeds back: compiler errors, test results, linter output. Anthropic's agent guide says an agent should get "ground truth" from the environment at each step, such as tool results or code execution (Anthropic, Building effective agents). With weak feedback (no tests, flaky tests, a test the agent wrote to pass), the loop converges on something that only looks finished. What makes agent output trustworthy is what made human output trustworthy: fast, independent, deterministic checks. The Claude Code docs make the same point: without a check the agent can run, "looks done" is the only signal, and you become the verification loop (Claude Code best practices).

Layer 3The machine

Under the agent, each tool call is an ordinary process: the harness calls fork and execve for pytest or go test, reads stdout and stderr through pipes, and waits with wait4. The model itself runs on accelerators where producing each output token requires reading the model weights from high-bandwidth memory; at tens of gigabytes of weights and a few terabytes per second of bandwidth, each decode step costs milliseconds, so a 2,000-token edit takes seconds to tens of seconds. A full test suite might take minutes. When the check costs far more than the generation, verification dominates wall-clock time, and faster tests (parallel runs, test selection, warm build caches) speed up the agent more than a faster model does.

Why it's built this way

  1. Generating plausible text is cheap for a model, but plausibility is not correctness, so every output needs an external check.
  2. External checks need a definition of correct, so someone has to frame the problem precisely before work starts.
  3. Checks cost real CPU time and human attention, so the cheapest effective check wins, and choosing it is a skill.
  4. Production differs from any test environment, so some failures are only found by diagnosis after release, and that loop has to be fast and well practised.

Diagram

flowchart LR
    S["Symptom or request"] --> F["Frame: problem + success metric"]
    F --> G["Generate (agent or human)"]
    G --> V{"Verify against oracle"}
    V -->|"fails"| D["Diagnose why"]
    D --> F
    D --> G
    V -->|"passes"| R["Release behind flag"]
    R --> P{"Production agrees?"}
    P -->|"no"| D
    P -->|"yes"| Done["Done"]

Worked example

A team is asked to "add rate limiting to the public API". Engineer A gives the request straight to an agent. In 10 minutes it produces an in-memory token bucket with passing unit tests. Review takes 40 minutes. In staging, behind 6 instances, clients get 6 times their limit because each instance keeps its own counters. The Redis rework costs an hour of agent time plus 90 minutes of review, and a clock-skew edge case found in production a week later costs a 2-hour investigation. Totals: about 1.5 hours generating, 2.5 verifying, 2 diagnosing, zero framing.

Engineer B spends 20 minutes framing: "Limit each API key to 100 requests per second across all instances; over-limit returns 429 with Retry-After; p99 overhead under 2 ms; fail open if the limiter store is down." That includes a question to the product owner about fail-open. The statement already rules out per-instance counters. B writes a load test firing 1,000 requests per second for one key through 6 instances, asserting about 100 succeed per second, then asks the agent to build against it. Generation 30 minutes, verification 45 minutes, no production diagnosis.

Same agent, same model. Twenty minutes of framing saved about four hours.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
Prompt and reviewFastest start, low ceremonyReview catches only what the reviewer thinks to look forSmall, reversible, well-tested areas
Frame, oracle, then generateClear target, agent converges faster, cheap reviewUp-front time, needs domain knowledgeAnything user-facing, stateful or hard to revert
Human writes, agent assistsDeep understanding retainedSlower generationNovel core logic the team must own deeply
Agent writes, independent agent reviewsCheap second opinionCorrelated blind spots if both share a model and contextAs an extra layer, never the only check

Pitfalls in production

Leadership lens

Check yourself

  1. Why can faster code generation make a team slower overall?

    Show answer

    If verification and diagnosis capacity stays the same, more generated changes queue for review and more defects reach production, so time saved in generation is spent on rework and incidents.

  2. What is an oracle, and why decide on it before generating code?

    Show answer

    An oracle is the independent way you judge correctness, such as a test, property or reference system. Choosing it first stops you from accepting whatever the generator produced as the definition of correct.

  3. Why is an agent's own test suite weak evidence that its code is right?

    Show answer

    The tests come from the same understanding of the task as the code, so a misread requirement shows up in both and they agree with each other.

  4. The METR study found developers felt faster but measured slower. What should a Tech Lead take from that?

    Show answer

    Self-reported productivity is unreliable, so measure phase durations and outcome metrics before changing process or headcount based on perceived gains.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →