Engineering Atlas › System Design Fundamentals

Running experiments: A/B testing plumbing

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Assignment

The unit id is hashed with an experiment-specific salt into, say, 10,000 buckets. Buckets map to variants by weight. The same user always lands in the same bucket, so assignment needs no database. To run many experiments at once, the platform uses layers: experiments in one layer split the traffic exclusively (a user is in at most one of them), while different layers use different salts so assignments are independent. Conflicting changes to the same page go in the same layer. In Google's design a layer owns a subset of system parameters, no parameter may belong to two layers, and domains (slices of traffic) can nest inside layers, so each request sits in at most one experiment per layer (Tang et al., KDD 2010).

Exposure logging

A user assigned to a checkout test who never reaches checkout should not count. The SDK emits an exposure event at the point the variant is actually used. Analysis includes only exposed users, which keeps the signal from being diluted.

Metric computation

A daily (or hourly) job joins exposure events with outcome events by unit id and computes per-user aggregates: conversions, revenue, sessions, latency. Per-variant means, variances, and confidence intervals follow. Precomputed per-user rows make each new analysis a cheap aggregation.

Statistics

For a conversion metric, a two-proportion z-test compares rates. Variance reduction techniques such as CUPED use each user's pre-experiment behaviour as a covariate; on Bing it cut variance by about 50 percent, the same power with half the users or half the duration (Deng et al., WSDM 2013). Gains depend on how well the pre-period predicts the metric, and new users have no pre-period at all. Sequential testing methods allow monitoring results continuously without inflating false positives, which fixed-horizon tests do not. "Always valid" p-values were built for exactly this and run in a commercial platform on hundreds of thousands of experiments (Johari et al.); the price is wider intervals at any fixed sample size (GrowthBook docs).

Skewed metrics such as revenue need more users before the normal approximation holds. One rule of thumb from Bing is at least 355 times the squared skewness per variant when skewness exceeds 1 (Kohavi et al., KDD 2014); capping extreme values is the usual fix.

Health checks

Before reading results, check the sample ratio. If a 50/50 test shows 50.8/49.2 on a million users, something is broken: a redirect that drops users in one arm, a bot filter, or a crash. A chi-square test flags this. It is not rare: about 6 percent of Microsoft experiments in one study showed an SRM, with causes ranging from assignment bugs to telemetry loss to bot filters that removed the most engaged treatment users (Fabijan et al., KDD 2019). Platforms add their own common causes, such as assigning on the server but logging exposure on the client, or hashing on a different id than the analysis joins on (GrowthBook docs). Guardrail metrics such as latency, errors, and unsubscribes stop a test that wins its target metric by hurting something else.

Down the stack

Layer 1The contract

The platform promises that assignment is deterministic, that assignments in different layers are statistically independent, and that each exposed unit is counted once, in the variant it actually saw. The scorecard promises an effect estimate with a confidence interval at a stated significance level, valid only if units are independent, the sample ratio matches the design, and the analysis plan was fixed in advance.

Layer 2The mechanism

Assignment is hash(salt + unit_id) mod buckets, with salts per layer. Google's overlapping experiment design adds domains and layers so that thousands of experiments share traffic without collisions. The SDK emits exposure events of the form (unit_id, experiment_id, variant, timestamp), deduplicated to the first exposure. A daily job builds a per-unit table of metric sums. From it each variant needs only sufficient statistics: count, sum, and sum of squares, from which mean and variance follow without rescanning raw events. Ratio metrics such as clicks per page view are not averages of independent rows, so their variance comes from the delta method (Deng et al., 2018). CUPED adjusts each unit's value by theta times its pre-period value, with theta equal to the covariance of the two divided by the pre-period variance.

Layer 3The machine

A billion events a day at about 200 bytes each is 200 GB of raw logs, which columnar Parquet with dictionary encoding typically shrinks to 20 to 40 GB. Joining exposures to events is a shuffle: both sides are hash-partitioned by unit id across the cluster, so the job is bound by network and disk spill, not CPU. Bots with millions of events land in one partition and create a straggler task that holds up the whole job, so bot filtering happens before the join. Computing variance as sum of squares minus square of sum in floating point can lose most significant digits when the mean is large relative to the spread; production code uses centred or Welford-style accumulation.

Why it's built this way

Storing billions of assignments would add a lookup to every request, so assignment is a hash. Analyses are rerun many times, so the expensive join happens once and later work uses small aggregates. Page views from one user are correlated, so the unit of analysis is the user and ratio metrics need special variance handling. Shuffles are the dominant cost, so data is partitioned by unit id once and reused.

Diagram

flowchart LR
  U[User request] --> A["Assign: hash(salt, user) to bucket"]
  A --> V{Variant}
  V --> App[Render variant]
  App --> E[Exposure event]
  App --> M[Outcome events]
  E --> L[(Event lake)]
  M --> L
  L --> J[Daily join per user]
  J --> S["Stats: effect, CI, SRM"]
  S --> D[Scorecard]

Worked example

A signup page converts at 4.0 percent. The team wants to detect a 5 percent relative lift, to 4.2 percent, with 80 percent power at a 5 percent significance level.

The required sample per arm is roughly 16 x p(1-p) / delta squared, where delta is 0.002. The 16 comes from 80 percent power at 5 percent significance. For 90 percent power, Kohavi's guide uses a total of (4 r sigma / delta) squared for r variants and warns that the 4 can overestimate by about 25 percent (Kohavi et al.). That is 16 x 0.04 x 0.96 / 0.000004, about 154,000 users per arm, 308,000 total. At 40,000 new visitors a day that is 8 days. Round up to 14 days to cover two weekly cycles.

With CUPED reducing variance by 30 percent, a conservative figure for a signup page with many new users, the requirement falls to about 108,000 per arm, under 6 days of traffic, which still rounds to a full week.

Peeking matters: if the team checks a fixed-horizon test every day and stops at the first p below 0.05, the real false-positive rate climbs well above 5 percent; checked after every observation it reaches 26.1 percent (Evan Miller). Either commit to the horizon or use a sequential method.

Pipeline scale: 20 million daily users at 50 events each is a billion events a day, which collapses to 20 million per-user rows.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Hash-based assignmentStateless, fast, reproducibleChanging weights mid-test reshuffles users unless handled carefullyDefault
Stored assignmentsCan change allocation without reshuffle, easy auditLookup on hot path, storageLong-running or regulated tests
Fixed-horizon testsSimple, well understoodPeeking invalidates themDisciplined teams, few tests
Sequential testsSafe continuous monitoring, early stopSlightly less power at the horizonMany tests, pressure to ship early
Switchback or cluster randomisationHandles network effects in marketplacesFewer units, more varianceTwo-sided markets, shared resources

Pitfalls in production

Leadership lens

Check yourself

  1. Why salt the hash with the experiment id?

    Show answer

    Without it, every experiment puts the same users in the same buckets, so treatment groups across experiments are correlated and effects can leak between tests.

  2. A 50/50 test shows 502,000 versus 498,000 users. Is that a problem?

    Show answer

    The chi-square statistic is 16, p is about 0.00006, so yes. A difference that large is very unlikely by chance and points to a bug in assignment, redirects, or logging.

  3. Why log exposure separately from assignment?

    Show answer

    Users assigned but never shown the change add noise with zero effect, diluting the measured lift and inflating the needed sample size.

  4. What does CUPED change about an experiment?

    Show answer

    It subtracts the part of each user's metric predicted by their pre-experiment behaviour, lowering variance and so the sample needed for the same power.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →