Engineering Atlas › System Design Interviews

Design a build and deploy pipeline

21 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional:

Non-functional:

Out of scope: the source code host itself, the test frameworks, and the internals of the deploy target (Kubernetes, serverless), which we drive through an API.

Back-of-envelope

Assume a large company or a mid-sized hosted CI service: 20,000 active developers.

Runs: each developer pushes about 10 times a day, giving 200,000 pipeline runs per day. Peaks follow working hours across time zones, so take 3x the average: 200,000 / 86,400 is 2.3 runs per second on average and about 7 per second at peak. Each run fans out to about 10 jobs, so 2 million jobs per day, 23 per second average, 70 per second peak.

Concurrency: with a median job duration of 6 minutes, average concurrency is 2,000,000 x 6 / 1,440 minutes, about 8,300 jobs at once. Peak is around 25,000 concurrent runners. Heartbeats at 25,000 / 15 s are about 1,700 writes per second, which a single well-tuned Postgres primary can absorb, but they are better kept in a cache with only lease expiries written to the database.

Logs: about 2 MB per job uncompressed, so 4 TB per day, roughly 600 GB per day after compression. With 90-day retention, about 55 TB.

Caches and artifacts: if each job uploads or downloads 300 MB of dependency and build cache, 2 million jobs move 600 TB per day through the cache, about 7 GB/s on average and over 20 GB/s at peak. This is the system's bandwidth bill, and it is why caches must sit in the same region (ideally the same zone) as runners. Stored artifacts with a 30-day TTL and content deduplication come to perhaps 1 to 2 PB.

Compute: 25,000 peak runners at 4 vCPUs is 100,000 vCPUs at peak. At a few cents per vCPU-hour, compute is by far the largest cost, which is why caching and test selection pay for themselves.

API

POST /v1/hooks/{provider}              (code host webhook; signed, carries delivery_id)
  -> 202

POST /v1/repos/{repo}/runs             body { ref, sha, trigger: "manual", inputs }
  -> 201 { run_id, status: "queued" }

GET  /v1/runs/{run_id}
  -> { run_id, sha, status, jobs: [ { job_id, name, status, attempt, started_at, ended_at } ] }

POST /v1/runs/{run_id}/cancel          -> 202
POST /v1/jobs/{job_id}/retry           -> 201 { job_id, attempt }

GET  /v1/jobs/{job_id}/logs?offset=N   (chunked; WebSocket upgrade for live tail)

# Runner protocol (runner -> scheduler, long poll)
POST /v1/runner/acquire   body { runner_id, labels, capacity: 1 }
  -> 200 { job_id, lease_token, lease_ttl_s, spec, oidc_audience } | 204 after 30 s
POST /v1/runner/jobs/{job_id}/heartbeat   body { lease_token }  -> 200 | 409 (lease lost)
POST /v1/runner/jobs/{job_id}/complete    body { lease_token, result, outputs }

# Deploy
POST /v1/deployments  body { environment, artifact_digest, strategy: "canary", steps: [5, 25, 100] }
  -> 201 { deployment_id, status: "waiting_for_lock" | "running" }

Runners only make outbound calls, so self-hosted runners can sit behind a firewall.

Data model

runs (Postgres, partitioned by repo_id hash, primary key run_id): repo_id, sha, ref, trigger, delivery_id (unique per provider, the idempotency key for webhooks), concurrency_group, status, created_at. One row per pipeline execution.

jobs (same shard as the run): job_id, run_id, name, needs (array of job names), labels, status (blocked, queued, leased, running, succeeded, failed, cancelled), attempt, lease_owner, lease_token (monotonic integer), lease_expires_at, cache_key. An index on (labels_hash, status, queued_at) serves dequeue. Keeping jobs with their run makes "job finished, unblock its dependents" one single-shard transaction.

environments and deploy_locks (small, global): one row per (repo, environment) with holder_deployment_id and fencing_token. A deploy must hold the lock; the token is passed to the deploy target so a stale deployer is rejected.

log chunks (object storage, key logs/{run_id}/{job_id}/{attempt}/{seq}): runners upload 64 KB to 1 MB chunks; a small index of chunk offsets lives in the jobs row or a key-value store. Live tail is served from a short-lived buffer before chunks are persisted.

content-addressed store (object storage, key cas/{sha256}): artifacts, cache archives and build outputs. An action cache maps cache_key -> list of output digests. Content addressing makes uploads idempotent and deduplicates across repos for free.

artifact attestations: for each release artifact, a signed provenance document (which commit, which pipeline, which runner image) stored next to the digest, following SLSA. The current spec is v1.2, which adds a Source track to the Build track (SLSA).

High-level design

flowchart LR
  VCS["Code host"] -->|webhook| WH["Webhook ingest"]
  WH --> Q[["Event queue"]]
  Q --> ORC["Orchestrator"]
  ORC --> DB[("Runs and jobs DB")]
  ORC --> CHK["Status reporter"]
  CHK --> VCS
  RUN["Ephemeral runners"] -->|"acquire, heartbeat"| SCH["Job dispatcher"]
  SCH --> DB
  AS["Autoscaler"] --> DB
  AS --> RUN
  RUN --> CAS[("Cache and artifact store")]
  RUN --> LOGS[("Log store")]
  RUN -->|OIDC token| CLOUD["Deploy target"]
  ORC --> DEP["Deploy controller"]
  DEP --> CLOUD
  DEP --> MET["Metrics for canary analysis"]

Write path: the code host delivers a signed webhook. Ingest verifies the signature, inserts delivery_id into a dedupe table, and enqueues the event, returning 202 within a few hundred milliseconds so the host does not retry. The orchestrator fetches the pipeline file at that exact SHA (never from the default branch, which would let a later commit change an earlier run), expands matrices into jobs, and writes the run and its jobs in one transaction. Jobs with no needs go to queued; the rest are blocked. If the run has a concurrency group (for example pr-4512), older runs in that group are cancelled in the same transaction.

Execution path: an idle runner long-polls acquire with its labels. The dispatcher picks the oldest queued job matching those labels with SELECT ... FOR UPDATE SKIP LOCKED LIMIT 1, sets it to leased with a new lease token and a 60-second expiry, and returns the job spec and a short-lived token. The runner restores caches, runs the steps, streams logs, heartbeats every 15 seconds, uploads outputs to the content-addressed store, and calls complete with its lease token. The orchestrator marks the job done, moves dependents whose needs are all satisfied to queued, and posts a commit status. A runner that disappears stops heartbeating; a reaper moves expired leases back to queued with attempt + 1, and any late complete carrying the old token is rejected.

Deploy path: a deploy job does not hold cloud keys. It requests an OIDC token from the CI system whose claims name the repo, branch and environment, and exchanges it with the cloud provider for credentials that last minutes. The deploy controller acquires the environment lock, rolls out the artifact digest in canary steps, compares error rate and latency with the baseline after each step, and rolls back automatically on regression.

Deep dives

Scheduling and the runner fleet

Push versus pull: a push scheduler must track every runner's address and health and handle the case where it sends a job to a runner that just died. Pull inverts that: runners ask for work only when idle, so a dead runner simply stops asking. The cost is a cheap long-poll connection per idle runner.

Isolation: shared long-lived runners leak state between jobs (a poisoned cache directory, a leftover credential, a background process). Ephemeral runners, one VM per job, destroyed after, remove that class of bug and attack. Full VMs take 20 to 60 seconds to boot, which would blow the 15-second start target, so the autoscaler keeps a warm pool sized from the recent queue arrival rate per label, for example the 95th percentile of arrivals over the next two minutes. MicroVMs such as Firecracker boot in well under a second and restore from snapshots, which shrinks the pool you need. GitHub's own guidance for self-hosted fleets matches this: register runners with --ephemeral so each takes exactly one job and is then deregistered, and do not autoscale persistent runners (GitHub docs).

Fairness: one team's 500-way test matrix must not starve everyone else. Dequeue per label is ordered by a per-tenant fair-share score (running jobs divided by quota) and then by age, not purely first-in-first-out. Hard quotas cap any tenant at a fraction of the fleet.

Choice: pull-based ephemeral runners with leases and fencing tokens, a warm pool per label, and weighted fair queuing across tenants. The same lease mechanism appears in Design Job Scheduler and Handling Long-Running Tasks.

Build avoidance: caching and test selection

The cheapest job is one that does not run. A build step is a function of its inputs: source files, dependency lockfile, toolchain image digest, environment variables and the outputs of the steps it depends on. Hash all of them into a cache key, Merkle style, where each step's key includes its dependencies' keys. If the key is already in the action cache, download the outputs instead of running the step. A change to one library file then changes only that library's key and the keys of things that depend on it, and the rest of a monorepo is reused. This is how Bazel's remote cache and remote execution work, with an action cache of results and a content-addressable store of output files (Bazel docs), and the practice task implements the key computation and the resulting plan.

Correctness depends on hermeticity: if a step reads something not in its key (the current time, a network fetch, an unpinned base image), the cache returns stale results. Coarse cache keys such as hash(lockfile) for a dependency directory are a pragmatic middle ground: they cannot be wrong about code, only about what is installed.

Test selection goes further: run only tests whose dependency closure includes a changed file, plus a periodic full run to catch what the graph misses. Google's TAP system does this at the scale of its whole monorepo (Google Research).

Keeping main green and deploying safely

Two changes can each pass against an old base and break when combined. A merge queue fixes this: approved pull requests enter a queue, the system builds the speculative merge of each one on top of the ones ahead of it, and lands them in order only if green. To keep throughput up, it tests several speculative merges in parallel and, when one fails, drops that change and rebuilds the ones behind it. GitHub's merge queue builds these speculative merges on temporary gh-readonly-queue/{base_branch} branches, triggers CI with a separate merge_group event (a required check that only listens to pull_request never reports, and the merge fails), and caps concurrent builds at between 1 and 100 (GitHub docs). Uber's SubmitQueue is the large-scale version: its 2025 refinements cut CI resource use by about 53% and P95 wait time by about 37% across Uber's Go, iOS and Android monorepos (arXiv).

Deploys must be serialized per environment, so the deploy controller takes a lease-based lock with a fencing token, and the deploy target rejects writes carrying an older token. The artifact is identified by digest (image@sha256:...), so what was tested is byte for byte what ships, and rollback is redeploying the previous digest. Credentials come from OIDC federation, scoped by claims such as environment:production, so a pull request from a fork, which runs without those claims, cannot obtain production access even if it changes the pipeline file. Write trust policies against the exact subject format: GitHub's classic subject looks like repo:octo-org/octo-repo:environment:prod, but repositories created after 15 July 2026 default to an immutable format that includes owner and repository IDs, so a policy copied from an older repo may silently never match (GitHub OIDC docs). See Canary Releases and Rollbacks and Immutable Infrastructure for the rollout mechanics.

Down the stack

Layer 1The contract

Each accepted webhook delivery_id creates at most one run. Each job ends in exactly one terminal state, but its steps may execute more than once (at-least-once execution), so jobs must be safe to repeat; deploy steps achieve this by being declarative ("make production run digest D") rather than imperative. Only the holder of the current lease token can complete a job or write its outputs as final. A cache hit for key K returns outputs that some successful execution with key K produced. At most one deployment per environment is in progress, enforced by a fencing token checked at the target.

Layer 2The mechanism

Dequeue is a Postgres transaction (SKIP LOCKED is documented for exactly this multi-consumer queue table (Postgres docs)): SELECT job_id FROM jobs WHERE labels_hash = $1 AND status = 'queued' ORDER BY priority, queued_at FOR UPDATE SKIP LOCKED LIMIT 1, then UPDATE ... SET status = 'leased', lease_token = lease_token + 1, lease_expires_at = now() + interval '60 s'. SKIP LOCKED lets hundreds of dispatcher threads dequeue concurrently without blocking on each other's rows. Completion is a conditional update WHERE job_id = $1 AND lease_token = $2; zero rows updated means the lease was lost and the runner discards its result. Unblocking dependents is a second statement in the same transaction that decrements a pending_needs counter on child jobs and flips those that reach zero to queued. The action cache follows the Bazel Remote Execution API's split between an action cache (ActionResult keyed by action digest) and a content-addressable store (blobs keyed by SHA-256 digest), so writes of blobs are idempotent and the action entry is written last.

Layer 3The machine

Starting a job is dominated by bytes, not CPU. A shallow git clone --depth=1 of a large monorepo still moves hundreds of MB; a persistent mirror on the runner image plus git fetch of the delta cuts that to a few MB. Pulling a 1 GB container image means downloading compressed layers at perhaps 200 to 500 MB/s from a same-region registry and decompressing gzip on one core at roughly 100 to 300 MB/s, so 5 to 10 seconds, which is why runner images are pre-baked into the VM snapshot. A 2 GB dependency cache tarball takes about 10 seconds to download and extract and then creates tens of thousands of small files, each an inode allocation and directory entry update, so restore can be bound by metadata operations on the runner's disk rather than by bandwidth. Firecracker's specification promises 125 ms or less from the InstanceStart API call to the guest's /sbin/init, with under 5 MiB of VMM memory overhead (Firecracker spec), against tens of seconds for a full cloud VM.

Why it's built this way

  1. Jobs run untrusted code, so each runs in a throwaway VM, which makes runners unreliable by design and forces leases, heartbeats and retries.
  2. Retries mean steps can run twice, so outputs are content addressed and deploys are declarative, which makes duplicates harmless.
  3. Fresh VMs start empty, so every job would re-download and rebuild the world; content-addressed caching keyed by input hashes turns that into a lookup.
  4. Cache traffic is petabytes per day, and cross-region bandwidth is expensive, so caches and runners live in the same region.
  5. The pipeline can reach production, so credentials are short-lived and scoped to claims about the run, not stored as long-lived secrets anyone with write access could read.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Long-lived shared runnersWarm caches, fast start, cheapState leaks between jobs, weak isolationTrusted internal repos with tight budgets
Ephemeral VM per jobStrong isolation, reproducibleCold caches, boot latency, needs warm poolsMulti-tenant or open-source workloads
Coarse cache keys (lockfile hash)Simple, works with any toolRebuilds whole steps on small changesMost repos without a hermetic build system
Hermetic build graph with remote cacheRebuilds only what changed, shareable across machinesMigration cost, every input must be declaredLarge monorepos with slow builds
Merge queueMain stays green under high merge ratesAdds queue latency, needs parallel speculationMany merges per hour into one branch
Direct merge after PR checksLowest latencySemantic conflicts break mainSmall teams, low merge rate

Leadership lens

Check yourself

  1. A runner's VM dies mid-job, then a replacement finishes the retried job. Later the original runner's delayed complete call arrives. Why does it not overwrite the result?

    Show answer

    The reaper moved the expired lease back to queued with a new, higher lease token, and completion is a conditional update WHERE job_id = $1 AND lease_token = $2. The stale token matches zero rows, so the old runner's result is discarded.

  2. Why does the design choose pull-based ephemeral runners over a push scheduler with long-lived shared runners?

    Show answer

    Pull means a dead runner simply stops asking for work, and a fresh VM per job removes state leaks such as poisoned cache directories or leftover credentials. The costs are cold caches and boot latency, which is why the autoscaler keeps a warm pool per label sized from recent queue arrivals.

  3. Estimate peak concurrent jobs and explain why cache traffic forces caches to sit next to runners.

    Show answer

    2 million jobs a day at a 6-minute median is about 8,300 concurrent on average and about 25,000 at peak. At 300 MB of cache per job that is 600 TB per day, over 20 GB/s at peak, so caches must be in the same region, ideally the same zone, as the runners.

  4. A non-hermetic step stores a bad output under a cache key that keeps hitting. Why does this happen, and how is it contained?

    Show answer

    The step read something not in its key, such as the current time, a network fetch or an unpinned base image, so the key no longer captures its inputs. Include toolchain image digests in keys, allow per-key invalidation, and never let untrusted pull-request jobs write caches that protected-branch builds read.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →