Engineering Atlas › System Design Interviews

Delivering video and audio at scale

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Encoding and the ladder

A source file (say a 4K master) is decoded and re-encoded into renditions such as 240p at 300 kbit/s up to 4K at 15 Mbit/s or more. Each rendition uses the same keyframe (IDR frame) positions so segments line up across renditions and the player can switch quality at any segment boundary without a glitch. Apple's authoring rules are a common baseline: an IDR every 2 seconds, every segment starting with one, and segments of nominally 6 seconds (Apple HLS authoring spec). Services tune the ladder per title or even per scene: a cartoon needs far fewer bits than a grainy action film for the same perceived quality.

Packaging and manifests

The encoded streams are cut into segments (fragmented MP4 under the CMAF format is now common to both HLS and DASH). A master playlist lists the renditions with their bandwidth, resolution and codecs; each media playlist lists the segments in order. Audio and subtitles are separate tracks with their own playlists, so a player can combine any video rendition with any language.

Delivery

The player fetches the master manifest, picks a starting rendition (usually a conservative one), then fetches segments one by one over HTTP from a CDN. Popular titles stay hot at the edge; large services also place caching appliances inside ISP networks so the bytes never cross the wider internet. Netflix's Open Connect works with over a thousand ISPs, gives the appliances to qualifying ISPs at no charge, and fills them nightly (Netflix Open Connect). Signed URLs or tokens limit access, and DRM (Widevine, FairPlay, PlayReady) encrypts segments with keys delivered by a separate license server.

Adaptive bitrate in the player

The player keeps a buffer of downloaded but unplayed media, typically tens of seconds for on-demand video. Throughput-based ABR estimates bandwidth from recent segment downloads and picks the highest rendition that fits with a safety margin. Buffer-based ABR picks quality from buffer level: low buffer means low quality, full buffer means high quality. Netflix tested this on millions of users and found capacity estimates unnecessary in steady state but still needed at startup, while the buffer is filling (Huang et al., SIGCOMM 2014). Production players mix both and add hysteresis so they do not flap between rungs; the FESTIVE algorithm, for example, uses a harmonic mean over the last 20 segments and moves only one rung at a time (Timmerer et al.).

Live

A broadcaster pushes a stream (RTMP, SRT or WebRTC) to an ingest server, which transcodes in real time into the ladder and publishes segments as they complete. WebRTC ingest now has a standard handshake: WHIP sends the SDP offer in one HTTP POST and ends the session with DELETE (RFC 9725). Latency is roughly the segment duration times the number of segments the player buffers, plus encode and CDN propagation; classic HLS with 6-second segments often sits at 15 to 30 seconds behind live. Low-latency HLS and DASH publish partial segments and use blocking playlist reloads to get to a few seconds. Parts are shorter than many guides suggest: Apple recommends a one-second part target, and at least three times the P95 round trip to the server, so that clients are not drowned in requests (same Apple spec). Sub-second interactivity needs WebRTC, which gives up CDN-friendly caching.

How it shows up in interviews

Triggers: design YouTube, Netflix, Twitch, Spotify, or a video course platform. A strong answer separates the upload and transcoding pipeline from playback, explains segments, manifests and ABR, puts a CDN in front with numbers on egress, and handles live separately if asked. Probes include: why not stream one big file, what happens when bandwidth drops mid-play, how many renditions and why, how you handle a new viral video that is not yet cached, and how you get live latency down.

Down the stack

Layer 1The contract

The server side promises immutable, independently decodable segments at stable URLs, each starting with a keyframe, and a manifest that accurately lists them; for live, a playlist that only grows and advances its media sequence number. The player promises continuous playback as long as average throughput exceeds the lowest rung, at the best quality it can sustain, with a startup time target (often a second or two) and a rebuffering ratio target (stall time as a fraction of watch time). Nothing promises a constant quality; the contract is "keep playing, as well as possible".

Layer 2The mechanism

HLS (RFC 8216) master playlists carry #EXT-X-STREAM-INF:BANDWIDTH=...,RESOLUTION=...,CODECS=... lines, each followed by a media playlist URI. Media playlists carry #EXT-X-TARGETDURATION, #EXT-X-MEDIA-SEQUENCE, and an #EXTINF:<duration> line before each segment URI; on-demand lists end with #EXT-X-ENDLIST, while live lists keep sliding. RFC 8216 itself predates low-latency HLS; EXT-X-PART-INF and blocking reloads live in its intended successor, still an Internet-Draft at revision 22 in May 2026 (IETF draft). DASH uses an XML MPD with Period, AdaptationSet and Representation elements and usually a SegmentTemplate such as $RepresentationID$/$Number$.m4s. In the browser, the player downloads segments with fetch and feeds bytes into Media Source Extensions with SourceBuffer.appendBuffer, so the video element plays a stream assembled by JavaScript. iPhones only gained this path with Safari 17.1 in 2023, through Managed Media Source (WebKit), a variant in which the browser may evict buffered media and tells the player when to fetch (MDN). The transcoding pipeline splits the source at keyframes into chunks, runs encoders such as x264 or x265 per chunk per rendition on a worker fleet, then concatenates and packages, which turns hours of serial encoding into minutes.

Layer 3The machine

A 1080p rendition at 5 Mbit/s is 625 KB per second, so a 4-second segment is about 2.5 MB, which TCP delivers as roughly 1,700 segments of 1,460-byte payload. With a 40 ms RTT, a fresh connection needs several round trips of slow start before the window is large enough, which is why players reuse connections (HTTP/2 or HTTP/3) and why the first segment is a low rung. On edge servers, serving is a page-cache-to-NIC problem: hot segments live in RAM, sendfile avoids copying them through user space, and kernel TLS keeps encryption in the kernel. Netflix has described its FreeBSD-based Open Connect appliances serving on the order of hundreds of gigabits per second per box using these techniques. Encoding is the opposite: CPU-bound, with a modern encoder at slow presets running well below real time per core for 4K, so encode cost scales with renditions times duration times preset effort.

Why it's built this way

Networks vary from second to second, so quality must be chosen per segment by the client that sees the network. HTTP caches are everywhere and cheap, so video is shaped into small immutable HTTP files. Switching requires aligned keyframes, so all renditions share a GOP structure. Encoding is expensive and done once, while delivery happens millions of times, so it pays to spend more CPU per title to save bandwidth on every view.

Diagram

flowchart LR
    UP["Upload source"] --> SPLIT["Split at keyframes"]
    SPLIT --> ENC["Parallel encoders (one job per chunk per rung)"]
    ENC --> PKG["Package CMAF segments and manifests"]
    PKG --> ORIGIN["Origin storage"]
    ORIGIN --> CDN["CDN edges and ISP caches"]
    CDN --> PLAYER["Player"]
    PLAYER -->|"measures throughput and buffer"| ABR{"ABR picks rung"}
    ABR -->|"next segment URL"| CDN

Worked example

A course platform serves 2 million viewing hours per day. Average delivered bitrate is 3 Mbit/s because many viewers watch on laptops at 720p or 1080p.

Egress: 2 million hours x 3,600 seconds x 3 Mbit/s = 21.6 billion megabits per day, about 2.7 PB per day, roughly 250 Gbit/s on average with an evening peak perhaps twice that. With a 95% CDN hit ratio, the origin serves about 5% of that, around 12 Gbit/s on average. Cutting the average bitrate 20% with per-title encoding saves about 540 TB of egress per day, which is why encoding effort pays for itself.

Storage: 50,000 hours of content with a six-rung ladder averaging 2.5 Mbit/s per rung stores 50,000 x 3,600 x 15 Mbit = 2.7 billion megabits, about 340 TB, small compared with daily egress.

Playback: segments are 4 seconds, the player targets a 30-second buffer, and starts at 750 kbit/s. On a 20 Mbit/s home connection, the first 4-second segment (375 KB) arrives in a few hundred milliseconds including connection setup, playback starts, and the player climbs one rung per segment to 1080p within about 20 seconds.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Short segments (2 s)Faster switching, lower live latencyMore requests, slightly worse compressionLive or variable networks
Long segments (6 to 10 s)Fewer requests, efficient encodingSlower adaptation, higher live latencyOn-demand with stable networks
Fixed ladderSimple pipelineWastes bits on easy contentSmall catalogue
Per-title or per-scene ladderLower bandwidth for the same qualityMore encode compute and complexityLarge audiences where egress dominates
LL-HLS or LL-DASHFew seconds of live latency over CDNsMore requests, tighter tuningSports, live events
WebRTCSub-second latencyPoor CDN fit, costly at large scaleInteractive streams, calls

Pitfalls in production

Leadership lens

Check yourself

  1. Why is streaming built on segments over HTTP rather than a single long stream?

    Show answer

    Segments are cacheable, immutable files that any CDN can serve, and each segment boundary is a chance for the player to change quality.

  2. Why must keyframes align across renditions?

    Show answer

    A segment must decode on its own, and switching quality mid-stream needs the next segment in any rendition to start with a keyframe at the same timestamp.

  3. Why does classic HLS live streaming lag 15 to 30 seconds?

    Show answer

    Players buffer several segments of several seconds each, and segments are published only when complete, so delay accumulates by segment duration.

  4. Why does a player prefer a harmonic mean of throughput samples?

    Show answer

    It weights slow samples heavily, so a single dip pulls the estimate down and the player avoids picking a rung it cannot sustain.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →