Engineering Atlas › System Design Interviews

Moving large files reliably

13 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why not through the app server

An app server that proxies a 4 GB upload holds a connection and buffers for minutes, ties up a worker or thread, doubles network traffic (client to app, app to storage), and loses everything if it is redeployed mid-upload. Object stores are built for this traffic; your servers are not. So the app server's job shrinks to authorization and bookkeeping.

The upload flow

  1. The client calls POST /uploads with file name, size and content type. The server checks quota and permissions, creates a metadata row with status PENDING, and starts a multipart upload in object storage, receiving an upload ID.
  2. The server returns pre-signed URLs, one per part (or issues them in batches on request). Each URL authorizes a PUT of one part until it expires, and it can be replayed any number of times before then. S3 checks expiry when a request starts, and a URL signed with temporary role credentials dies when that session does, by default after an hour (AWS docs), so clients must be able to ask for fresh URLs mid-upload.
  3. The client uploads parts in parallel, typically 3 to 8 at a time, each 8 to 64 MB. It records which parts succeeded and their ETags or checksums.
  4. On a network drop, the client asks the server (or the store) which parts exist and uploads only the missing ones.
  5. The client calls POST /uploads/{id}/complete with the part list. The server completes the multipart upload, verifies size and checksum, and moves the row to UPLOADED.
  6. An event on completion triggers post-processing: virus scanning, thumbnailing, transcoding, indexing. The row moves to READY when done.

Chunking for sync and deduplication

Sync products go further: they split files into content-defined or fixed-size blocks and name each block by its hash; Dropbox's storage keys blocks of up to 4 MB by their SHA-256 (Dropbox Tech). Uploading a file becomes "which of these block hashes do you not already have?", so editing one page of a large document uploads one block, and two users uploading the same file store it once. The file's metadata becomes an ordered list of block hashes.

Serving downloads

For public or widely shared files, put a CDN in front of the store. For private files, the server checks permissions and returns a pre-signed GET URL or a signed CDN URL with a short expiry. Clients use Range headers to resume broken downloads and to fetch parts in parallel; the store answers 206 Partial Content, and an If-Range validator makes sure the bytes still belong to the same version (RFC 9110).

How it shows up in interviews

Triggers: design Dropbox or Google Drive, design YouTube or Instagram upload, design an attachment service for email or chat, design a backup service. A strong answer keeps bytes off app servers, uses pre-signed multipart uploads with resumability, defines the metadata state machine, verifies integrity end to end, and handles post-processing asynchronously. Interviewers probe: how a client resumes after a laptop sleeps, how you stop someone uploading 1 TB against a 10 GB quota, what happens to abandoned uploads, how you deduplicate, and how a shared link to a private file stays secure.

Down the stack

Layer 1The contract

A pre-signed part URL promises that exactly that operation (method, bucket, key, part number, sometimes content length and checksum) is allowed until expiry; anyone holding the URL can use it, so it is a bearer token. Multipart upload promises atomic visibility: the object does not exist until complete succeeds, and then it appears whole. Parts can be re-uploaded; the last successful upload of a part number wins. Your API promises that a file in READY has the declared size and checksum and has passed scanning. Range requests promise byte-exact slices of an immutable object, which is why objects should be immutable and new versions get new keys.

Layer 2The mechanism

In Amazon S3, CreateMultipartUpload returns an UploadId; each UploadPart takes a PartNumber from 1 to 10,000 and a body of at least 5 MiB (except the last) and at most 5 GiB, and returns an ETag; CompleteMultipartUpload takes the ordered list of part numbers and ETags. The object size limit is now 48.8 TiB, not the 5 TB that older material quotes, and ListParts returns at most 1,000 parts per page, so a resuming client must paginate (AWS docs). The resulting ETag of a multipart object is the MD5 of the concatenated part MD5s plus -N, not the MD5 of the file, so use explicit checksums. S3 now attaches a CRC64NVME checksum by default when the client sends none, and CRC-based algorithms support a "full object" checksum on multipart uploads, so you can check one whole-file value without tracking part boundaries; SHA-256 works only as a composite of part checksums (AWS docs). Incomplete uploads keep their parts, and their storage charges, until aborted, which is what the AbortIncompleteMultipartUpload lifecycle rule is for. Pre-signed URLs carry a SigV4 signature in the query string (X-Amz-Signature, X-Amz-Expires, X-Amz-Credential). Google Cloud Storage uses a resumable session URI instead, valid for a week, where the client sends chunks in multiples of 256 KiB with Content-Range and asks for the committed offset after a failure; Google suggests one chunk when possible, since each extra request adds latency and cost (Google Cloud docs). The open tus protocol standardises the same idea: HEAD returns Upload-Offset, and PATCH sends bytes starting at that offset. Its successor is an IETF working-group draft, still not an RFC at version 13 in October 2026, which adds an Upload-Complete header and a 104 interim response that hands the client a resume URL before the body is sent (IETF draft).

Layer 3The machine

Upload time is bound by the client's uplink. A 2 GB file over a 20 Mbit/s home uplink takes about 800 seconds even with perfect parallelism, so resumability matters more than speed. A single TCP connection's throughput is limited by its congestion window over RTT; on a 100 ms path, one connection with a 1 MB window caps near 80 Mbit/s, and one lost packet halves the window under classic congestion control, which is why parallel parts over several connections fill a fat pipe better. On the serving side, a well-built file server avoids copying bytes through user space: sendfile(2) moves data from the page cache to the socket inside the kernel (man7), and kernel TLS (kTLS) lets that work for HTTPS too. Storage servers erasure-code objects across disks and racks (for example splitting data into k data and m parity fragments), so a whole-object read touches many disks in parallel, and a 1 KB metadata row lives in a database built for small random reads.

Why it's built this way

Bytes and metadata have opposite access patterns (large sequential versus small random), so they live in different systems. Client networks are slow and flaky, so uploads must resume from parts instead of restarting. App servers are expensive per connection and redeployed often, so they hand clients a signed capability and step aside. Objects are immutable once complete, which makes caching, ranges and CDN distribution safe.

Diagram

sequenceDiagram
    participant C as Client
    participant A as Upload API
    participant DB as Metadata DB
    participant S as Object store
    C->>A: POST /uploads (name, size)
    A->>DB: insert PENDING row
    A->>S: CreateMultipartUpload
    A-->>C: upload id and pre-signed part URLs
    par parallel parts
        C->>S: PUT part 1
        C->>S: PUT part 2
    end
    C->>A: complete (part list, checksum)
    A->>S: CompleteMultipartUpload
    A->>DB: status UPLOADED
    S-->>A: event: scan and process

Worked example

A design-file sharing product has 5 million uploads per day with a median size of 40 MB and a p99 of 3 GB. Total ingest is about 200 TB per day.

If uploads went through app servers, average ingest of 200 TB per day is about 18 Gbit/s sustained, with daytime peaks of several times that, all entering and leaving app servers. With direct-to-store uploads, the API handles about 120 create and complete calls per second on average, each a few kilobytes.

Part size is 16 MB. A 3 GB file is 192 parts; with 6 in parallel over a 50 Mbit/s uplink it takes around 8 minutes. If Wi-Fi drops at minute 6, the client lists parts already uploaded (roughly 140) and resends only the rest, losing at most the 6 parts in flight, about 96 MB. A lifecycle rule aborts multipart uploads that are not completed within 7 days; before it existed, the team found tens of terabytes of orphaned parts quietly billed each month.

Integrity: the client computes a SHA-256 per part and for the whole file while reading it. The store verifies part checksums on receipt, and the server checks the whole-file checksum after completion before marking the file UPLOADED.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Proxy through app serversSimple, full control of bytesCostly, fragile, double bandwidthSmall files only (a few MB)
Single pre-signed PUTSimple, no app bandwidthNo resume, size limits per requestFiles up to tens of MB
Multipart with pre-signed partsParallel, resumable, very large filesMore client logic, orphan cleanupLarge uploads, unreliable networks
Content-addressed blocksDedup and delta syncHash index, garbage collection of blocksSync and backup products

Pitfalls in production

Leadership lens

Check yourself

  1. Why should part uploads be independent and retryable?

    Show answer

    Networks drop mid-transfer, and independent parts mean a failure costs one part rather than the whole file.

  2. Why not compare the multipart ETag with an MD5 of the file?

    Show answer

    The multipart ETag is derived from the parts, not the whole file's MD5, so integrity needs an explicit checksum computed the same way on both sides.

  3. What does content-addressed chunking enable?

    Show answer

    Uploading only blocks the server lacks, so edits send small deltas and identical content is stored once.

  4. Why do completed objects get new keys instead of being overwritten?

    Show answer

    Immutable keys make CDN caching and range resumes safe, since a cached or partially downloaded object can never change underneath.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →