Engineering Atlas › System Design Fundamentals

Automating build, test and release

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Integration

Developers push small changes to a short-lived branch or to trunk. A webhook triggers CI, which checks out the exact commit, restores caches, compiles, runs unit tests and static checks, and reports status to the pull request. Branch protection blocks the merge until required checks pass. Aim for a verdict in under ten minutes, the old Extreme Programming guideline that Martin Fowler's 2024 revision of his CI article still calls reasonable for most projects (Fowler). The same article is strict about what counts: everyone integrates into mainline at least daily, and running CI on long-lived feature branches is only semi-integration, because the branches still diverge from each other.

Build once, promote many

The build stage produces one artifact, usually a container image addressed by its content digest. Later stages pull that digest and never rebuild. Rebuilding per environment means testing one binary and shipping another, since a floating base image or dependency can change in between. Google goes further and makes builds hermetic: pinned tool and library versions, so two people building the same revision on different machines expect identical output (Google SRE book, ch. 8).

Stages as a DAG

Pipelines are graphs. Lint, unit tests, and scans run in parallel after compile; staging deploy waits for all of them. Jobs run on ephemeral runners and pass state only as declared artifacts, which makes them cacheable and retryable.

Delivery and deployment

After staging, the pipeline waits for approval (delivery) or proceeds automatically (deployment). The production deploy itself should be progressive and gated by health checks that can roll back without a human.

Supply chain

Record provenance: which commit, builder, and inputs produced this digest. Signing the artifact and verifying at deploy time stops an attacker who can push to the registry but cannot run the trusted builder. SLSA is the common yardstick for this; the current version is 1.2, released in November 2025, which added a Source Track for how code is written and reviewed alongside the Build Track (SLSA). Runners building fork PRs must never see production secrets, and GitHub warns that workflows triggered by pull_request_target can see secrets, so they must never check out the fork's code (GitHub docs).

Third-party CI steps are dependencies too. A tag like v4 can be moved to point at new code, so pin actions to a full commit SHA; GitHub's newer workflow dependency lockfile (.github/workflows/actions.lock, in public preview) records the exact commit of every action, including transitive ones (GitHub docs).

Down the stack

Layer 1The contract

A CI system promises a verdict per required check for an exact commit SHA, not for a branch name. A merge queue strengthens that: every commit that lands on trunk has passed checks on exactly the tree that landed, including other PRs merged ahead of it. GitHub's queue does this by building temporary gh-readonly-queue/<base> branches that stack each PR on the ones ahead; if one fails, it is removed and the later groups are rebuilt without it (GitHub docs). The build stage promises an artifact identified by a content digest, so two people asking for "release 412" get identical bytes. The deploy stage promises at most one rollout per environment at a time, and that a newer commit is never overwritten by an older one.

Layer 2The mechanism

A push produces a webhook event carrying the head SHA. The scheduler expands the pipeline file into jobs, and runners long-poll a queue to claim them. Caches are keyed by a hash of the inputs, for example deps- plus the SHA-256 of the lockfile, with prefix fallbacks when the exact key misses. Build tools such as Bazel go further: every action declares its inputs, command line and allowed environment variables, the action cache maps a hash of that definition to its results, and outputs live in a content-addressable store, so an unchanged library is never rebuilt anywhere in the company (Bazel docs). A container image is a JSON manifest that lists layer digests and a config digest (OCI image spec). Its own digest is the SHA-256 of that manifest, so changing any layer changes the identity. Deploy credentials come from OIDC: each job gets its own signed JWT with claims such as sub, repository, ref and job_workflow_ref, the cloud checks those claims against the role's trust policy, and returns an access token valid only for that job (GitHub docs).

Layer 3The machine

A fresh runner VM boots in 10 to 40 seconds. Restoring a 1 GB dependency cache from object storage at about 200 MB/s takes 5 seconds; installing the same packages from a registry means tens of thousands of small file writes and metadata operations, which is often slower than the download itself. Image layers are usually gzip tarballs, so a 300 MB layer costs both network time and single-threaded decompression on pull; the OCI spec also defines zstd-compressed layers, which decompress faster where registries and runtimes support them (OCI image spec). Integration tests that start PostgreSQL in a container spend much of their time in fsync; putting test databases on tmpfs or disabling fsync for throwaway instances can halve suite time.

Why it's built this way

Engineer waiting time costs far more than runner minutes, so pipelines trade compute for latency through parallel jobs. Most job time goes to moving bytes, so everything is cached by content hash. Content hashes give artifacts an identity, so promotion by digest becomes possible. Long-lived deploy keys stored in CI get leaked, so credentials are minted per job from a signed identity and expire quickly.

Diagram

flowchart LR
  A[Push or PR] --> B[Checkout + cache]
  B --> C[Compile]
  C --> D[Unit tests]
  C --> E[Lint + SAST]
  C --> F["Build image (digest)"]
  F --> G[Integration tests]
  D --> H[Merge gate]
  E --> H
  G --> H
  H --> I[Deploy staging]
  I --> J[Smoke + e2e]
  J --> K{Auto or approval}
  K --> L[Progressive prod deploy]
  L --> M{Health OK?}
  M -->|no| N[Auto rollback]
  M -->|yes| O[Done]

Worked example

A team of 40 engineers merges about 120 changes a day into a monorepo. The naive pipeline runs every test on every commit: 25 minutes on one runner. At 120 runs a day that is 50 runner-hours, and a developer waits 25 minutes per change.

Three changes fix this. Caching keyed on the lockfile hash removes about 6 minutes. Sharding tests across 8 runners turns 15 minutes into about 3.5 including startup. Affected-target analysis runs only tests downstream of changed packages, about 30 percent of the suite on a typical change.

The verdict now arrives in about 6 minutes. At 0.008 USD per runner-minute the old pipeline cost 120 x 25 x 0.008 = 24 USD a day; the new one is roughly 15 USD, because most commits run a fraction of the tests. The bigger win is human: 120 changes x 19 saved minutes is 38 engineer-hours of waiting per day.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Continuous delivery (manual prod gate)Human checkpoint, easy fit for regulated change controlBatches pile up behind the gate, larger risky releasesAudit rules need named approvers, or monitoring is immature
Continuous deploymentSmallest batches, fastest feedback, no release toilNeeds strong tests, automated rollback, feature flagsMature observability and a culture of small changes
Monorepo with affected-target buildsOne pipeline, atomic cross-service changesNeeds a build graph tool, cache infrastructureMany services share libraries
Per-service pipelinesSimple, isolated ownershipDrift between pipelines, cross-service changes are hardIndependent teams with few shared libraries

Pitfalls in production

Leadership lens

Check yourself

  1. Why should staging and production deploy the same image digest instead of each rebuilding from the same commit?

    Show answer

    A rebuild can pull different base images or dependency versions, so the tested artifact and the shipped artifact differ. Promoting one digest guarantees what passed tests is byte-identical to what runs in prod.

  2. A merge queue batches 5 PRs and the batch fails. What does the queue do next, and why is that better than testing each PR alone?

    Show answer

    It isolates the failing PR (by bisecting the batch, or as GitHub does, from the stacked per-PR builds), ejects it, and merges the rest. In the common all-green case it pays for one test run instead of five, while still keeping trunk green.

  3. What separates continuous delivery from continuous deployment, and what prerequisite does the second need?

    Show answer

    Delivery keeps every commit releasable but a human triggers production; deployment ships automatically. It needs automated health gates and rollback, because no human is watching each release.

  4. Why must jobs for pull requests from forks never receive production secrets?

    Show answer

    The fork author controls the code the job runs, so any secret in scope can be exfiltrated by a malicious test or build script.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →