Engineering Atlas › Problem Solving & Incidents (AI era)

Shipping Agent Code Safely: Small Batches, Flags and Rollbacks

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why volume changes the game

If each change carries a small chance of causing a problem, more changes mean more problems in absolute terms, unless each change gets safer. A team that went from 20 to 80 merged pull requests a week with agents, at the same 5 percent change failure rate, now has 4 bad changes a week instead of 1. The answer is not to slow down. It is to make each change smaller, more observable and cheaper to undo. DORA's 2024 research found that AI adoption raised individual productivity but also hurt delivery stability and throughput, and concluded that small batches and robust testing remain crucial (DORA 2024).

Small batches

A 50-line change with one purpose can be reviewed properly, tested specifically and reverted cleanly. A 3,000-line agent change that refactors, adds a feature and bumps dependencies cannot. Ask agents for one concern per pull request; split their work into a sequence of small, independently safe steps (add the column, write to it, backfill, read from it, remove the old one). Trunk-based development, with short-lived branches merged at least daily, keeps changes small by default.

Deploy is not release

A feature flag lets code reach production switched off. Deploying then becomes routine and low risk, and releasing becomes a separate, controlled act: turn it on for staff, then 1 percent of users, then more. If something goes wrong, turning the flag off takes seconds, much faster than building and deploying a revert. Flags have costs: each one is a branch in the code, and old flags become confusing dead code, so give every flag an owner and a removal date.

Progressive delivery

Whether by flag percentage or by canary deployment (new version on a few servers), the pattern is the same:

  1. Release to a small slice.
  2. Wait long enough to collect enough requests to judge.
  3. Compare the slice's error rate and latency with the baseline's over the same period.
  4. Advance if healthy, hold if there is not enough data, roll back if worse beyond tolerance.

Automating step 4 matters most when changes are frequent: humans watching dashboards for 80 rollouts a week will miss things. Amazon's pipelines do this with waves: each starts with a "one-box" deployment serving at most about 10% of requests, then a bake time (at least an hour after one-box, 12 hours after the first regional wave), and an alarm the deployment system watches so it can roll back automatically (AWS Builders' Library). Google's SRE workbook adds that fleet-wide metrics hide canary problems: a 20% error rate on a 5% canary shows up as only about 1% overall, so break metrics out by canary and control (SRE workbook).

Rollback readiness

Every change should have an answer to "how do we undo this in five minutes?". Code changes: flag off or redeploy the previous version. Schema changes: expand-then-contract, so the old code still works against the new schema. Data changes: backups and a tested restore. Changes that cannot be undone quickly, such as sending emails or deleting data, belong in the gatekeeper zone with human approval.

Down the stack

Layer 1The contract

The delivery system promises that every production change is small, traceable to a reviewed pull request, released progressively with automated health checks, and reversible within minutes. Its health is measured by change failure rate and time to restore service, reported per team and per author type (human or agent). DORA now describes five metrics: change lead time, deployment frequency, failed deployment recovery time (the successor to MTTR), change fail rate and deployment rework rate (DORA).

Layer 2The mechanism

Flag systems evaluate rules per request: hash a stable identifier such as the user ID into a bucket from 0 to 99, and enable the feature if the bucket is below the rollout percentage, so each user sees consistent behaviour as the percentage grows. Canary analysis compares metrics between the canary and a baseline group running the old version at the same time, which controls for traffic changes during the day. Keep the metric window no longer than the canary, since an hourly metric muddies a 30-minute canary, and never compare against yesterday's numbers (SRE workbook). A typical rule: roll back if the canary's error rate exceeds the baseline's by more than a relative tolerance (say 20 percent) and an absolute floor (say 0.1 percentage points), and only judge once each side has a minimum number of requests, such as 1,000, to avoid reacting to noise.

Layer 3The machine

Rolling back a deployment usually means the orchestrator shifts traffic back to processes running the old binary. In Kubernetes, the previous ReplicaSet is scaled up and the new one scaled down; with connection draining, load balancers stop sending new requests to old pods and wait for in-flight ones to finish, often for a configured grace period such as 30 seconds. Flags avoid even that: the flag service pushes a new rule set to clients over a streaming connection or polling, and each process re-evaluates on its next request, typically within seconds. That speed difference is why flags are the preferred first responder.

Why it's built this way

  1. More changes at the same failure rate means more failures.
  2. Smaller changes fail less often and are easier to diagnose and undo.
  3. Separating deploy from release turns most rollbacks into a configuration change.
  4. Progressive exposure limits how many users a bad change can hurt before it is caught.
  5. At high volume only automated comparison and rollback keep up.

Diagram

stateDiagram-v2
  [*] --> Dark: deploy with flag off
  Dark --> Pct1: release 1%
  Pct1 --> Pct5: healthy
  Pct5 --> Pct25: healthy
  Pct25 --> Pct100: healthy
  Pct1 --> Pct1: not enough data
  Pct5 --> Rollback: worse than baseline
  Pct1 --> Rollback: worse than baseline
  Pct25 --> Rollback: worse than baseline
  Rollback --> Dark: flag off
  Pct100 --> [*]: remove flag later

Worked example

A team of 10 engineers uses agents heavily and merges about 120 pull requests a week. Their change failure rate is 6 percent, so about 7 changes a week cause a problem, and their median time to restore is 50 minutes because most fixes need a revert, build and deploy.

They make three changes. First, a pull request size limit of 400 changed lines with an exception process, plus agent instructions to split work into steps. Median pull request size drops from 380 to 140 lines. Second, every user-facing change ships behind a flag, released 1, 5, 25, 100 percent with automated canary checks: at least 2,000 requests per stage, rollback if errors exceed baseline by 25 percent relative and 0.1 points absolute. Third, an expand-and-contract rule for schema changes.

After two months, the change failure rate, counting only failures that reached more than 5 percent of users, falls to 2 percent. The median time to restore falls to 4 minutes, because most rollbacks are automated flag flips at the 1 or 5 percent stage. Deployment frequency rises, because small, flagged changes no longer wait for a "safe" release window.

In the wild

Trade-offs

PracticeStrengthsWeaknessesChoose when
Feature flagsInstant rollback, gradual releaseCode complexity, stale flagsUser-facing changes
Canary deploymentsCatches infrastructure and performance issuesNeeds traffic and good metricsServices with steady traffic
Blue-greenFast full switch and switch backDouble capacity during switchStateful or hard-to-canary systems
Big-bang releaseSimpleLarge blast radius, slow rollbackRarely; internal tools with few users
PR size limitsReviewable, bisectable changesFriction on legitimately big changesTeams with agent-generated volume

Pitfalls in production

Leadership lens

Check yourself

  1. Why does higher change volume require smaller changes rather than more review?

    Show answer

    Review time does not scale with volume, but smaller changes are each less likely to fail, easier to review and faster to undo, so total risk falls without slowing delivery.

  2. Why hash a stable user ID to decide flag exposure instead of choosing randomly per request?

    Show answer

    So each user sees consistent behaviour, and users in the 1 percent stay included as the rollout grows to 5 and 25 percent.

  3. Why compare the canary with a baseline running at the same time rather than with yesterday's metrics?

    Show answer

    Traffic mix and load vary over time; a concurrent baseline controls for those differences so the comparison isolates the change.

  4. Why is a flag flip usually a faster rollback than redeploying?

    Show answer

    It changes configuration that running processes pick up within seconds, while a redeploy needs a build or image pull, pod scheduling and connection draining.

Sources and further study

Primary

Systems docs

Books

Free resources

Open in the interactive app: practice, quiz, mock interview →