Engineering Atlas › System Design Interviews

Design a feature flag service

19 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope

Back-of-envelope

Assume 50,000 server processes, 2,000 projects and 5,000 flags per environment.

API

# Control plane (humans and automation)
PUT  /v1/projects/{p}/envs/{e}/flags/{key}      # body: flag definition, header If-Match: <flag version>
POST /v1/projects/{p}/envs/{e}/flags/{key}/kill # body: {"on": true, "reason": "INC-4411"}
GET  /v1/projects/{p}/envs/{e}/audit?flag=checkout&limit=50
POST /v1/projects/{p}/envs/{e}/rollback         # body: {"to_version": 4812}

# Data plane (SDKs)
GET  /sdk/stream?since=4812        # text/event-stream: put (snapshot) and patch (delta) events
GET  /sdk/ruleset                  # full snapshot, ETag = version, served from CDN or object store
POST /sdk/events                   # aggregated exposure counts
{ "key": "checkout-v2", "salt": "k3x9", "enabled": true, "kill": false,
  "rules": [ { "attribute": "country", "values": ["ID", "SG"], "variation": "on" } ],
  "rollout_bp": 2000, "rollout_variation": "on",
  "fallthrough": "off", "off_variation": "off" }

rollout_bp is in basis points (2000 = 20%). A stream event looks like event: patch / id: 4813 / data: {"upsert":[...],"delete":[...]}; the id is the ruleset version.

Data model

The environment version, not the flag version, is the unit the SDK tracks. A single integer lets an SDK say "I have 4812" and a relay answer with exactly the changes since.

High-level design

flowchart LR
    UI["Admin UI / API"] --> CP["Control plane"]
    CP --> DB[("Flags + audit DB")]
    CP --> PUB["Publisher"]
    PUB --> OBJ[("Snapshot in object storage")]
    PUB --> RELAY["Stream relays"]
    OBJ --> CDN["CDN"]
    RELAY --> SDK["SDK in app: ruleset in memory"]
    CDN --> SDK
    SDK --> EVAL["evaluate(flag, context)"]
    SDK --> EV["Aggregated exposure counts"]
    EV --> TEL["Telemetry pipeline"]
    BOOT["Bootstrap file on disk"] --> SDK

Write path. An engineer saves a flag. The control plane checks permissions, validates the definition (known operators, rollout between 0 and 10000, referenced segments exist), then in one transaction updates flags, appends the audit row and increments the environment version. The publisher writes the new snapshot to object storage first and then emits a patch with that version to the relays, so a client that misses the patch can always fetch a snapshot at least as new.

Read path. At startup an SDK loads the bootstrap file if present, then fetches the snapshot from the CDN, then opens the stream with since=<version>. Evaluation never touches the network. A background poll every 30 to 60 seconds sends If-None-Match: <version> so a missed event is repaired within a minute even when the stream silently stalled.

Deep dives

1. Getting changes to 50,000 processes

Fan-out is the real scale problem: relays hold the 50,000 connections. After a relay restarts, 20,000 SDKs reconnect at once, so each waits a random delay (capped exponential backoff with jitter), because synchronised retries turn one failure into a cascade (Google SRE book).

2. Stable percentage rollouts and targeting

A rollout must satisfy three properties: the same user always gets the same answer, increasing the percentage only adds users, and different flags select statistically independent groups.

The salt matters. If every flag hashed only the user id, the same 5% of users would be the guinea pigs for every experiment. With a per-flag salt each flag draws an independent 5%, and changing a salt on purpose reshuffles everyone, which restarts an experiment cleanly.

Multivariate flags split the 10,000 buckets into consecutive ranges, for example A 0 to 3333, B 3334 to 6666, C the rest. Targeting rules run before the rollout and are evaluated in order, first match wins; a user-id allow-list is just a rule. The result is returned with a reason (KILLED, OFF, RULE:2, ROLLOUT, FALLTHROUGH, MISSING_FLAG), which is exactly what OpenFeature's resolution details carry, so debugging "why did this user get that?" never needs a guess (OpenFeature spec).

The unit you hash is a product decision: user id for logged-in features, device id for anonymous traffic, organisation id when a whole tenant must see one experience. A context with no id cannot be bucketed, so the fallthrough must be safe.

3. Fail-static and safe changes

The control plane will be down sometimes, so every layer needs a defined behaviour:

  1. Control plane down. SDKs and relays keep the last version; nothing running changes.
  2. Relay or stream down. The SDK polls the CDN snapshot, then falls back to its in-memory copy.
  3. Cold start with everything down. Load the bootstrap file from the image or local disk, else use the coded default at the call site.

This is static stability: the system keeps doing what it was doing when a dependency is impaired, though it stops seeing updates (AWS Builders' Library). Each call site chooses its code default deliberately, and it should be the safe, old behaviour.

The harder half is making a bad change survivable, because flags change production behaviour everywhere in seconds. Facebook's configuration system treats such changes as code: reviewed, validated, canaried and distributed through one pipeline, with thousands of changes a day and trillions of checks (SOSP 2015 paper). Apply that here:

Down the stack

Layer 1The contract

  • Evaluation is deterministic: the same ruleset and context give the same (value, reason) in every SDK and language. That needs a hash with a fixed specification, not a language's built-in hash.
  • Freshness is bounded: a connected SDK reflects a published version within 5 s at p99. A disconnected one serves its last version and reports stale.
  • Monotonic versions: a process never moves backwards to an older version, and a rollback is a new, higher version whose content equals an old one.
  • Never throws: any internal error returns the call-site default with an ERROR reason.
  • Audit completeness: every published version has exactly one audit entry naming an actor.

Layer 2The mechanism

  • Stream framing: SSE lines event: patch, id: 4813, data: {...} then a blank line, plus a : keepalive comment every 15 to 30 s so idle proxies stay open.
  • Resume: on reconnect the SDK sends Last-Event-ID: 4813. The relay replays from a ring buffer of the last few thousand patches, or sends a put event with a fresh snapshot when the gap is too large.
  • Snapshot endpoint: ETag equal to the version, short max-age, served from a CDN so a fleet restart hits storage once per edge.
  • Hash: 32-bit FNV-1a or MurmurHash3 over UTF-8 bytes of flag_key.salt.unit_id, taken mod 10000. FNV-1a is a loop of XOR and multiply by 0x01000193; Murmur3 distributes better for very small rollouts.
  • Swap: the SDK holds one immutable Ruleset object behind an atomic pointer. Evaluation reads the pointer once and uses that snapshot throughout, so a patch arriving mid-evaluation can never produce a half-old, half-new answer.
  • Pre-evaluated clients: for browsers, an edge function evaluates all flags for one context and returns a small JSON map, so rules never reach untrusted code.

Layer 3The machine

  • Evaluation cost: a hashed lookup of the flag (about 20 to 50 ns), a short rule loop over strings, and a 20-byte FNV hash (about 20 ns) put a check around 100 to 300 ns, all on data that stays in L1 and L2 cache.
  • Parse cost and GC: decoding 3 MB of JSON takes roughly 10 to 30 ms and allocates tens of MB. Do it only when the version changes, off the request path, not on every poll.
  • Network: a 1 KB patch is one TCP segment (MSS about 1,460 B), so delivery is one relay-to-SDK half round trip, a few milliseconds in-region. A fresh TLS handshake costs 1 to 2 round trips, so SDKs keep the stream open.
  • Relay sockets: 20,000 streams per node means 20,000 file descriptors, and the kernel's epoll_wait loop wakes on writes only.
  • Disk: the bootstrap file is written with write-to-temp, fsync, then rename so a crash leaves either the old or the new file, never half of one.

Why it's built this way

  1. A flag check is on every request, and a network round trip costs 0.5 ms in-region against 0.2 microseconds in memory, so evaluate locally.
  2. Local evaluation means every process holds a copy, so copies drift. Version numbers and a poll backstop make drift bounded and detectable.
  3. A pure function of ruleset and context lets the hash be the assignment, so no per-user state is needed and the answer is reproducible on any machine.
  4. A flag change can alter behaviour everywhere in seconds, so safety (validation, staged rollout, audit, kill switch) is more important than raw speed.
  5. The control plane is the least available part, because it is the one humans edit. Therefore the data plane must not depend on it to keep running.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Local evaluation of a streamed rulesetSub-microsecond, survives control-plane outage, attributes stay privateRuleset in every process, staleness, SDK per languageBackend services, the default
Remote evaluation API per checkOne implementation, rules never leave the serverLatency and availability on every request, per-check costBrowsers and phones, via an edge cache
Hash-based bucketingNo state, sticky, monotonicCannot move one user without a rule, unit choice mattersPercentage rollouts
Fail-static to last goodApp unaffected by control-plane outageNew kill switches do not arriveAlways, with a break-glass path for urgent changes

Leadership lens

Check yourself

  1. Why must the hash include a per-flag salt, and what happens if you change the salt later?

    Show answer

    Without a salt, the same low-numbered buckets are chosen by every flag, so the same users receive every experiment and results are correlated. Changing the salt moves every user to a new random bucket, which restarts the experiment but also reshuffles who sees the feature.

  2. A relay restarts and 20,000 SDKs reconnect at the same instant. What goes wrong and what do you do?

    Show answer

    The relay and snapshot origin face a synchronised burst that can trip its limits and fail again in lockstep. SDKs reconnect after a randomised exponential backoff, resume with Last-Event-ID, and fetch snapshots from a CDN, while relays cap connections and return Retry-After.

  3. The control plane has been down for six hours and a feature is misbehaving. What can an operator still do?

    Show answer

    Apps keep serving the last ruleset, so nothing breaks, but the normal UI path is dead. A break-glass path writes the kill switch directly to the publisher or object store snapshot, which relays and the polling backstop then deliver.

  4. Why store 50,000 SDK copies of 3 MB instead of asking a service per check, given 100 million checks a second?

    Show answer

    A remote check would be 100 million requests/s of 0.5 ms round trips and would put the flag service on every request path. Local evaluation costs memory (3 MB per process) and bounded staleness, a far better trade.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →