Engineering Atlas › System Design Interviews

Design an online coding judge

20 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

Requirements

Functional

Non-functional

Out of scope: the editor itself, discussion forums, premium billing, interview mode, and post-contest plagiarism detection (mentioned briefly).

Back-of-envelope

Round assumed numbers, not LeetCode's figures.

API

GET  /api/problems/{slug}               -> { title, statement_html, examples, languages, limits }

POST /api/submissions
     { problem: "two-sum", language: "cpp17", code, mode: "submit" | "run",
       custom_input?, contest_id?, client_token: "uuid" }
     -> 202 { submission_id: "s_8812", status: "queued" }

GET  /api/submissions/{id}              -> { status: "queued" | "running" | "done",
                                             verdict?, passed?, total?, runtime_ms?, memory_kb?,
                                             failing_case? (examples only), compile_log? }
WS   /ws/submissions/{id}               <- { status: "running", case: 12, total: 50 } ... { status: "done", ... }

GET  /api/contests/{id}/ranking?page=1  -> { rows: [ { rank, user, score, finish_s, problems: [...] } ] }

client_token makes the POST idempotent: a double click or a retry after a timeout returns the same submission_id. The verdict comes back over a WebSocket or, as a fallback, by polling with a backoff of 0.5, 1 and 2 s (see Long Polling). The hidden test that failed is never returned, only its index, so tests cannot be scraped one at a time.

Data model

problems: key slug. Statement, limits per language (a time multiplier for slower runtimes), test_set_version. Read-mostly and cached at the CDN; changes are rare and editorial.

test_sets: key (problem_id, version). Object storage key, SHA-256 of the archive, count of cases and a checker type (exact match, token match, floating-point tolerance or a custom checker program). Versioned so a fixed test never silently changes old verdicts and every worker can verify it has the right bytes.

submissions: key submission_id, a time-ordered 64-bit id. Columns: user_id, problem_id, language, code_ref, test_set_version, status, verdict, passed, runtime_ms, memory_kb, contest_id, created_at. Partitioned by user_id, because the main read is "my history for this problem"; a secondary index on (contest_id, created_at) serves contest processing.

user_problem: (user_id, problem_id) -> best_verdict, first_ac_at. A small table that powers the green check marks on every problem list without scanning history.

idempotency: (user_id, client_token) -> submission_id with a 24-hour TTL.

contest standing: in Redis, a sorted set per contest whose score encodes rank order, plus a hash per contestant holding per-problem attempts and accept times. The durable truth is the submissions table; Redis is a rebuildable view (see Redis).

High-level design

flowchart LR
  U["Browser"] --> CDN["CDN (problem pages)"]
  U --> API["API service"]
  API --> DB[("Submissions DB")]
  API --> QC["Contest queue"]
  API --> QP["Practice queue"]
  API --> QR["Run queue"]
  QC --> W["Judge workers"]
  QP --> W
  QR --> W
  W --> SB["Sandbox per execution"]
  W --> TS[("Test sets (object store + SSD cache)")]
  W --> RS["Result service"]
  RS --> DB
  RS --> LB[("Leaderboard (Redis)")]
  RS --> PS["Push gateway (WebSocket)"]
  PS --> U

Submit path. The API authenticates the user, checks the per-user rate limit, deduplicates on client_token, writes the code to storage and the submission row as queued, and enqueues a small message (submission_id, problem, version, language) on the contest, practice or run queue. It returns 202 immediately.

Judge path. A worker leases a message, fetches the code, makes sure the test set version is in its local cache (verifying the hash), and starts a sandbox. It compiles once, then runs each test case with limits and compares output with the checker, stopping at the first failure. It reports the verdict to the result service and only then acknowledges the message.

Result path. The result service writes the verdict with a conditional update (status != done), so a duplicate report from a retried job is ignored. It updates user_problem, updates the contest standing if the submission belongs to a contest, and pushes the verdict to the browser.

Deep dives

1. The sandbox

The options form a ladder of isolation strength against start-up cost. A plain process with setrlimit and a chroot is fast but one kernel bug away from the host. Containers add namespaces (PID, network, mount, user, IPC), cgroup v2 limits (memory.max, cpu.max, pids.max) and a seccomp-bpf filter allowing only the system calls a compiled program needs. A user-space kernel such as gVisor intercepts system calls before they reach the host kernel; its own docs warn of higher per-syscall overhead and poor performance for syscall-heavy workloads (gVisor docs). A microVM such as Firecracker gives each execution its own guest kernel and boots in about 125 ms. Seccomp on its own is not the sandbox: the kernel documentation says plainly that syscall filtering only shrinks the kernel's attack surface and must be combined with other isolation (Linux seccomp docs).

I would run each submission in a Firecracker microVM (or gVisor) holding a nsjail-style inner sandbox: defence in depth, because this attack surface is the whole internet. Inside, no network interface exists, the root filesystem is read-only, the only writable space is a small tmpfs, stdout is capped (say 64 MB) so an infinite print loop cannot fill the disk, and pids.max stops fork bombs: once the limit is reached, fork() and clone() fail with EAGAIN (kernel pids controller docs). Tools such as nsjail package exactly this combination of namespaces, cgroups, rlimits and a seccomp policy (nsjail). The judge itself must also treat everything inside the sandbox as hostile: any host-side code that writes into or chowns the sandbox directory must refuse to follow symlinks, a lesson Judge0 learned the hard way (see In the wild). Compilation runs in the same sandbox with its own limits, because a compiler fed hostile templates or macros can be made to consume all memory too.

To hide the start-up cost, workers keep a pool of pre-booted, language-specific sandboxes. One sandbox serves exactly one submission and is then destroyed, never reused, so nothing written by one user is visible to the next.

2. Queues, fairness and the contest surge

Three queues with separate consumer pools keep the traffic classes apart. Run requests are interactive and small, Submits during a contest decide rankings, and practice Submits can wait a few extra seconds. A contest surge then cannot starve the practice user who is debugging, and a flood of Runs cannot delay contest verdicts. Within a queue, per-user rate limits (for example 1 Submit per 5 s per problem) and a per-user in-flight cap stop one script from filling the fleet.

Because contest start times are known a week ahead, scale ahead rather than react. Twenty minutes before the start, the autoscaler raises the judge fleet to the forecast peak (last contest's peak plus margin), and workers pre-load test sets for the contest's problems, which are hidden until the start. Reactive autoscaling covers the error in the forecast. When the queue still grows, the system degrades on purpose: it shows the queue position, delays practice Submits, and keeps the contest queue's oldest-message age as the alerting signal (see Absorbing Traffic Spikes).

A job that crashes a worker is retried after its lease expires. A job that crashes three workers goes to a dead-letter queue and gets a "System Error" verdict, which carries no penalty. That rule is what makes at-least-once job delivery safe for contestants.

3. Fair timing and the leaderboard

Wall-clock time punishes a submission for sharing a host with a noisy neighbour, so the time limit is enforced on CPU time read from the cgroup's cpu.stat, with a wall-clock cap of about three times the limit to catch programs blocked on input. Judge hosts run one sandbox per dedicated physical core, with SMT siblings left idle and frequency scaling fixed, so the same program measures within a few percent on any host. Limits are set at two to three times the reference solution's time, and slower languages get a multiplier, so the verdict depends on the algorithm, not on noise.

For contests, ranking is by score descending, then by finish time ascending, where finish time is the time of the last accepted submission plus a fixed penalty for each wrong attempt on problems eventually solved (LeetCode uses a five-minute penalty). Encode both keys into one sorted-set score, for example score × 10^7 - finish_seconds, so ZRANGE ... REV returns rank order and ZREVRANK gives a user's rank in O(log n). (ZREVRANGE still works but has been deprecated since Redis 6.2 (Redis docs).) Two details of the encoding matter. Scores are 64-bit doubles, exact only for integers up to 2^53, so a composite score must stay well inside that range; and members with equal scores are ordered by their bytes, so exact ties fall back to alphabetical user IDs unless you break them deliberately (Redis ZADD docs). Only accepted submissions change the set; wrong attempts only touch the contestant's hash. Pages of the top ranks are cached for a few seconds, which turns 1,700 reads/s into a handful of Redis calls. After the contest, the board is recomputed from the submissions table, which is the source of truth, and plagiarism checks run before ratings change.

Down the stack

Layer 1The contract

A 202 means the submission is durably recorded and will receive exactly one final verdict. Given the same code, language and test set version, the verdict is the same on any worker, and runtime varies by no more than about 10%. Hidden test content never leaves the judge. A submission can read only its own input and write only its own output.

Layer 2The mechanism

The worker leases a job with a visibility timeout of about 60 s and extends it while running. For each test, the sandbox supervisor creates a child cgroup with memory.max set to the limit, pids.max at 64 and cpu.max restricting it to one core. It forks, applies setrlimit limits (RLIMIT_CPU as a hard backstop, RLIMIT_FSIZE for output), installs the seccomp filter and execves the binary with input on stdin. The supervisor waits with a wall-clock timer. Afterwards it reads cpu.stat for usage_usec and memory.peak for peak usage, and checks memory.events for an oom_kill, the count of processes in the cgroup killed by any OOM killer (cgroup v2 docs). Classification order matters. CPU usage above the limit is TLE even if the process was then killed by a signal. An OOM kill is MLE. A non-zero exit or a fatal signal is a Runtime Error. Only a clean exit gets its output compared, by the problem's checker.

Layer 3The machine

fork plus execve of a static C++ binary costs about 1 ms; a JVM start is 100 to 300 ms and a Python interpreter about 20 ms, which is why language limits differ and why warm interpreters are tempting (they are not allowed here, since a warm process shares state between runs). Firecracker's specification promises 125 ms or less from InstanceStart to the guest's init process and under 5 MiB of VMM memory overhead for a small microVM (Firecracker spec), which is what makes one VM per submission affordable at 1,000/s across a few hundred hosts. Reading cpu.stat gives microsecond accounting from the kernel scheduler, unaffected by the host's load. Test data is read from local NVMe through the page cache, so the second run of a popular problem reads it from RAM. The noisiest effects are cache and memory bandwidth sharing between cores, which is why hosts run fewer sandboxes than they have cores.

Why it's built this way

  1. The code is hostile, and kernels have bugs, so isolate with two layers and destroy the sandbox after one use.
  2. Isolation and compilation take seconds, which a request thread cannot hold, so accept, queue and judge asynchronously.
  3. Queues deliver at least once and workers die, so verdict writes are conditional and crashes earn a penalty-free System Error.
  4. Rankings must be fair, so measure CPU time on quiet, pinned cores with generous margins.
  5. Contest load is predictable, so capacity is provisioned before the spike rather than after it.

Failure modes

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Process with rlimits and chrootFastest start, simplestOne kernel bug from escapeTrusted code only
Container with seccomp and cgroupsFast, well understoodShares the host kernelLow-risk internal judges
gVisorSyscalls handled in user spaceSlower for syscall-heavy codeStrong isolation, container tooling
Firecracker microVMOwn guest kernel, about 125 ms bootMore hosts to manage, KVM neededPublic judges, untrusted code at scale
Synchronous judgingSimple APIHolds connections for seconds, fails under spikesPrototypes
Queue plus workersAbsorbs spikes, retries, prioritiesDelivery is at least once, needs idempotent writesProduction
WebSocket for verdictsInstant, live per-case progressConnection state at the edgeMost users
PollingWorks everywhereExtra requests and latencyFallback

Leadership lens

Check yourself

  1. A judge job crashes three workers in a row. What happens to it, and why does this rule make at-least-once delivery safe for contestants?

    Show answer

    After its lease expires it is retried, and after crashing three workers it goes to a dead-letter queue with a "System Error" verdict that carries no penalty. Combined with the conditional verdict write (status != done), a retried or duplicated job can never double-count a penalty.

  2. Why does the lesson wrap a nsjail-style sandbox inside a Firecracker microVM rather than using a container with seccomp and cgroups alone?

    Show answer

    A container shares the host kernel, so it is one kernel bug away from escape, which is unacceptable when the attack surface is the whole internet. A microVM gives each execution its own guest kernel and boots in about 125 ms, which is affordable, and the inner sandbox adds defence in depth.

  3. Roughly how much judge CPU is needed on an average day versus a contest peak, and what does that imply for capacity planning?

    Show answer

    About 3M Submits at 2.5 CPU-seconds plus 6M Runs at 1 CPU-second is 13.5M CPU-seconds a day, roughly 160 busy cores, while a contest peak of 1,000 executions per second at about 1.7 s each needs about 1,700 cores. The fleet must scale about 10x for an hour, and because contest start times are known, it is pre-scaled about twenty minutes ahead.

  4. Why is the time limit enforced on CPU time from the cgroup rather than wall-clock time, and what else keeps verdicts repeatable across workers?

    Show answer

    Wall-clock time punishes a submission for sharing a host with a noisy neighbour, while cpu.stat gives kernel-accounted CPU time unaffected by host load; a wall-clock cap of about three times the limit still catches programs blocked on input. Hosts also run one sandbox per dedicated core with SMT siblings idle and fixed frequency, limits are two to three times the reference solution, and versioned test sets keep inputs identical.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →