Engineering Atlas › System Design Fundamentals

Shipping code dark with feature flags

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The evaluation model

A flag has a key, variations, and ordered targeting rules: explicit user allowlists, attribute rules (country == "DE", plan == "enterprise"), then a percentage rollout, then a default. The SDK receives the full rule set, evaluates it in-process against a context object (user id, tenant, region, app version), and returns a variation. Vendors have moved from "users" to typed contexts, so one flag can roll out by user, by organisation, or by device, each hashed on its own key and kind (LaunchDarkly docs).

Pete Hodgson's names for the parts are useful in design review: the toggle point is the if in your code, the toggle router decides, and the toggle configuration is the data the router reads (martinfowler.com). OpenFeature, a CNCF incubating project, standardises the router's API so the vendor behind it can be swapped through a provider (OpenFeature).

Deterministic percentage rollout

Percentage rollouts hash a stable unit id together with the flag key, for example hash(flag_key + ":" + user_id), map it to a bucket from 0 to 9,999, and enable the flag if the bucket is below the rollout threshold. Including the flag key decorrelates flags, so the same 10 percent of users are not always first. Because the bucket is fixed, raising the rollout from 10 to 20 percent keeps the first 10 percent enabled and adds new users. LaunchDarkly, for example, uses 100,000 buckets, so a rollout can be set to three decimal places of a percent, and widening variation A from 50 to 70 percent simply adds buckets 50,001 to 70,000 (LaunchDarkly docs).

Delivery

A control plane streams flag definitions to SDKs, with polling as a fallback; changes propagate in about a second. Servers bootstrap from a local snapshot or relay proxy so a flag-service outage does not block startup. Browser and mobile SDKs receive evaluated values for one user, not the rules, to avoid leaking targeting data.

Defaults and failure

Every call site supplies a default used when the SDK has no data. Pick it deliberately: usually the old behaviour, but for a kill switch the safe default is "feature running".

Lifecycle

Each flag gets an owner and expiry. Report stale flags (100 percent for 30 days, or unevaluated for 30 days) and make removal part of done. Tools encode this per kind: Unleash expects release and experiment flags to live 40 days and operational flags 7, treats kill switch and permission flags as permanent, and marks a flag "potentially stale" once it outlives its type (Unleash docs). Hodgson goes further and suggests "time bombs": a test that fails once a flag passes its expiry date (martinfowler.com).

You cannot test every combination of 50 flags. Test the configuration you expect in production, the fallback with the new flags off, and optionally everything on (martinfowler.com).

Down the stack

Layer 1The contract

A flag SDK promises that evaluation is deterministic for a given rule-set version and context, never throws, and always returns a variation, falling back to the caller's default on any error. OpenFeature writes this down as a requirement: client methods must not throw, and abnormal execution returns the default with an error code and reason in the evaluation details (OpenFeature spec 1.4). Updates are eventually consistent: a change reaches most processes within a few seconds, with no ordering guarantee across flags and no atomic multi-flag change. A process that loses its connection keeps serving its last known rules.

Layer 2The mechanism

The SDK keeps an in-memory map from flag key to a parsed rule set with a version number. Updates usually arrive over Server-Sent Events: a put event carries the full data set on connect, patch events carry one flag with its new version, and delete events leave a versioned tombstone (LaunchDarkly Go SDK). The SDK applies a patch only if its version is higher than the stored one, which makes out-of-order delivery harmless. Evaluation walks prerequisites, individual targets, then rules made of clauses (in, startsWith, semVerGreaterThan), then a fallthrough rollout. One widely used SDK computes the bucket by joining flagKey.salt.contextKey, taking the first 15 hex digits of its SHA-1, dividing by 16^15 - 1, and comparing against weights expressed in units of 1/100,000; experiments can replace the key and salt with a fixed seed (LaunchDarkly evaluator). Exposure events are counted in memory and flushed in batches every few seconds.

Layer 3The machine

Two thousand flags at about 2 KB each is 4 MB of rules per process. An evaluation is a hash map lookup, a few string comparisons, and one hash over roughly 40 bytes, a few hundred nanoseconds. Updates swap a pointer to a new immutable map (copy on write), so readers never take a lock on the hot path. Each SSE stream is a long-lived HTTP connection, so 10,000 pods mean 10,000 open sockets on the flag service. After an outage they all reconnect and each pulls the full 4 MB payload, 40 GB in a burst, which is why SDKs reconnect with jittered exponential backoff and large fleets put relay proxies in each cluster, so thousands of local connections share one upstream stream (LaunchDarkly Relay Proxy). One Go SDK starts its reconnect delay at 1 second, caps it at 30, and applies 50 percent jitter (LaunchDarkly Go SDK).

Why it's built this way

Flags sit on hot paths, so evaluation must be local. Network partitions are normal, so stale but available beats failing. Messages can be reordered or replayed, so each flag carries a version. Fan-out to thousands of processes is expensive, so relays and backoff protect the control plane.

Diagram

flowchart LR
  UI[Flag console] --> CP[Control plane]
  CP -->|stream rules| SDK1["SDK in service A"]
  CP -->|stream rules| SDK2["SDK in service B"]
  SDK1 --> C{"Evaluate: allowlist, rules, hash bucket"}
  C -->|variation| APP[Code path]
  SDK1 --> S[(Local snapshot)]
  SDK1 -.->|exposure events| AN[Analytics]

Worked example

A service handles 15,000 RPS and each request checks 12 flags. That is 180,000 evaluations per second per fleet. In-process evaluation with a hash and a few comparisons costs about 1 to 2 microseconds, so the total is under half a CPU core across the fleet. A remote call per evaluation at 1 ms each would add 12 ms per request and 180,000 calls per second to a flag service, which is why remote evaluation is a design error.

Rollout of a new pricing page: employees on day 1, then 1 and 10 percent of users. On day 4 a bug appears for a legacy currency. A rule currency == "XYZ" -> off propagates in a second with no deploy. Full rollout follows on day 7, and the flag and old path are deleted the next sprint.

The cost side: 400 engineers creating 15 flags each per year and removing none gives 6,000 live flags, an untestable space. Expiry policies keep the count in the low hundreds. Microsoft's Azure DevOps team uses a blunter nudge: one XML file of flag definitions per service, so a growing file motivates its owners to delete old flags (Microsoft Learn).

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Config file flags in the repoSimple, reviewed, versionedNeeds a deploy or config push to changeFew flags, low change frequency
Hosted or self-run flag service with streaming SDKsInstant changes, targeting, audit logAnother critical dependency, vendor costMany teams, frequent releases
Database-backed homegrown flagsFull controlEasy to build remote-per-call evaluation and no auditStrong platform team, special needs
Branch-based releases (no flags)No runtime branchesLong-lived branches, painful merges, no kill switchVery small codebases

Pitfalls in production

Leadership lens

Check yourself

  1. Why hash the flag key together with the user id instead of the user id alone?

    Show answer

    Hashing the user id alone places the same users in the lowest buckets for every flag, so a small group would always get every new feature first. Including the flag key gives each flag an independent assignment.

  2. Why does deterministic hashing matter when raising a rollout from 10 to 20 percent?

    Show answer

    The users below the old threshold stay below the new one, so nobody who had the feature loses it, and only new users are added.

  3. What should happen if the flag service is down when a pod starts?

    Show answer

    The SDK should load the last known snapshot or a relay proxy copy and fall back to safe defaults otherwise, so the service starts and behaves predictably.

  4. How is a kill switch different from a release toggle?

    Show answer

    A kill switch is long-lived and meant for operators to shed or disable functionality in incidents; a release toggle is temporary and removed after rollout.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →