Engineering Atlas › System Design Fundamentals

How TLS secures a connection

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The TLS 1.3 handshake

The client sends ClientHello with supported cipher suites, a random value, and a key share: an ephemeral elliptic curve public key, usually X25519. The server replies with ServerHello containing its own key share. Both sides now compute the same shared secret with ECDHE, and every following handshake message is encrypted. The server sends its certificate chain and a CertificateVerify signature over the handshake transcript, proving it holds the private key. It finishes with a Finished MAC. The client checks the chain, the signature, and the hostname, sends its own Finished, and application data flows. Total: one round trip. The protocol was respecified in July 2026 as RFC 9846, a cleanup that obsoletes RFC 8446 and keeps the wire format compatible (RFC 9846). TLS 1.0 and 1.1 are formally deprecated and must not be negotiated at all (RFC 8996).

Forward secrecy

Because the key exchange uses ephemeral keys discarded after the handshake, stealing the server's long-term private key later does not decrypt recorded past traffic. TLS 1.3 made this mandatory by removing static RSA key exchange.

Certificate validation

The client builds a path from the leaf certificate through intermediates to a root in its trust store. For each link it checks the signature, the validity period, and that the issuer is allowed to act as a CA. For the leaf it checks that the hostname matches a Subject Alternative Name; the old Common Name field must not be used for this. Wildcards must be the whole leftmost label and cover exactly one label: *.example.com matches api.example.com but not example.com or a.b.example.com (RFC 9525, which replaced RFC 6125 in 2023).

Revocation is the weak spot, and the industry is moving away from it. Let's Encrypt stopped putting OCSP URLs in certificates in May 2025 and turned its OCSP responders off in August 2025, citing the privacy leak of telling a CA which sites each visitor opens; revocation is now published only as CRLs (Let's Encrypt). The other answer is short lifetimes. CA/Browser Forum ballot SC-081 cut the maximum public certificate lifetime from 398 days to 200 days on 15 March 2026, then 100 days in 2027 and 47 days in March 2029 (CA/B Forum, DigiCert).

Resumption and 0-RTT

A server can issue a session ticket. On reconnect the client presents it and skips certificate work. With 0-RTT the client sends application data in its first flight, but that data can be replayed by an attacker, so only idempotent requests should be allowed. In HTTP, a proxy that forwards early data adds Early-Data: 1, and a backend that won't risk a replay answers 425 Too Early so the client retries after the handshake (RFC 8470).

Mutual TLS

In mTLS the client also presents a certificate. Service meshes use it for workload identity, rotating certificates every few hours.

Down the stack

Layer 1The contract

TLS promises confidentiality and integrity of an ordered byte stream, and that the server holds the private key for a certificate valid for the requested hostname. It does not hide the server's IP address, the size and timing of messages, or the SNI hostname unless Encrypted Client Hello is used. ECH became a Proposed Standard in March 2026; it encrypts the inner ClientHello, including SNI and ALPN, to a key the client gets from DNS (RFC 9849). Data sent as 0-RTT early data has no replay protection. Truncation is detected only because each side sends a close_notify alert before closing.

Layer 2The mechanism

Data travels in records of at most 16 KB plaintext. Each record has a 5-byte header (content type 23 for application data, legacy version 0x0303, length) and a 16-byte AEAD tag. The per-record nonce is a static IV XORed with a 64-bit sequence number, so a reordered or replayed record fails authentication. Keys come from a schedule built on HKDF: an early secret, then a handshake secret mixed with the ECDHE output, then a master secret, from which client and server traffic secrets are derived. A KeyUpdate message ratchets traffic keys forward on long connections. Certificates are DER-encoded X.509 with a Subject Alternative Name extension for hostnames and basicConstraints CA:TRUE on issuers. The client's SNI extension tells the server which certificate to present, and the status_request extension asks for a stapled OCSP response.

Layer 3The machine

A ClientHello is typically 250 to 500 bytes, but a hybrid post-quantum key share such as X25519MLKEM768 adds about 1.2 KB and can push it over one TCP segment. That is now the common case, not an edge case: about 70% of browser traffic reaching Cloudflare uses hybrid ML-KEM key exchange (Cloudflare), so middleboxes that assume a one-packet ClientHello break. The server's first flight with a certificate chain is often 3 to 5 KB. A new TCP connection starts with an initial congestion window of 10 segments, about 14 KB (RFC 6928), so a bloated chain with an extra intermediate can cost an additional round trip. AES-GCM with AES-NI and carry-less multiply instructions runs at several GB/s per core. Kernel TLS lets the kernel encrypt records, so a server can sendfile straight from the page cache without copying data into user space. Record size matters for latency: a 16 KB record cannot be decrypted until all of it arrives, so servers use small records early in a connection and grow them later.

Why it's built this way

Asymmetric cryptography is slow and symmetric cryptography is fast, so the handshake uses the former only to agree on keys for the latter. Traffic gets recorded and keys get stolen later, so key exchange is ephemeral. TCP slow start penalises large first flights, so chains and certificates should be small. Middleboxes broke on new version numbers, so TLS 1.3 disguises its handshake as a TLS 1.2 session resumption.

Diagram

sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: ClientHello (ciphers, key share)
  S->>C: ServerHello (key share)
  Note over C,S: both derive handshake keys via ECDHE
  S->>C: Certificate chain (encrypted)
  S->>C: CertificateVerify + Finished
  C->>C: verify chain, hostname, signature
  C->>S: Finished
  C->>S: HTTP request (application keys)

Worked example

A public API terminates TLS on a fleet of proxies handling 40,000 new connections per second at peak.

Cost per full handshake: one X25519 key agreement on each side and one signature by the server. With an ECDSA P-256 certificate, a signature costs roughly 30 to 50 microseconds of CPU; an RSA-2048 signature costs around 1 ms. At 40,000 handshakes per second, RSA needs about 40 CPU cores just for signing, while ECDSA needs 1 to 2. Switching to ECDSA certificates (keeping an RSA cert for very old clients) saves dozens of cores.

Latency: on a 100 ms round trip, TLS 1.2 adds 200 ms before the request and TLS 1.3 adds 100 ms; 0-RTT removes even that for safe requests.

Bulk AES-GCM runs at several GB per second per core with hardware support, so the handshake, not bulk encryption, is where TLS CPU goes.

Certificate operations: with 90-day certificates on 300 domains, there are about 3 renewals a day. Let's Encrypt plans to move its default from 90 to 45 days, and public certificates cap at 47 days from 2029, so plan for about 7 renewals a day on the same fleet. Renewal must be automated through ACME, with alerts driven by the fraction of lifetime used rather than a fixed day count, since "14 days left" is a third of a 45-day certificate.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Terminate at edge load balancerCentral cert management, offloads CPUInternal traffic plaintext unless re-encryptedTrusted internal network, simple setups
End-to-end TLS or mTLS meshEncrypted and authenticated everywhereCert issuance and rotation at scale, proxy overheadZero-trust networks, compliance
ECDSA certificatesFaster handshakes, smaller certsA few very old clients lack supportDefault today
Short-lived certificates (days)Limits damage of key theft without revocationRequires reliable automationInternal PKI, meshes, and public sites via Let's Encrypt's opt-in 160-hour profile (Let's Encrypt)

Pitfalls in production

Leadership lens

Check yourself

  1. Why does stealing a server's private key not let an attacker decrypt past TLS 1.3 sessions they recorded?

    Show answer

    Session keys come from ephemeral ECDHE shares that were discarded after each handshake. The long-term key only signed the handshake; it never encrypted the session keys.

  2. Does *.example.com cover v2.api.example.com?

    Show answer

    No. A wildcard matches exactly one label in the leftmost position, so it covers api.example.com but not deeper subdomains.

  3. Why are many teams moving to short-lived certificates instead of relying on revocation?

    Show answer

    Revocation checking is unreliable and often soft-fails, so a stolen key stays usable. A certificate that expires in days limits exposure without depending on revocation.

  4. Where does most TLS CPU cost go on a busy proxy, and how do you reduce it?

    Show answer

    In handshake signatures and key exchange. Use ECDSA certificates, session resumption, and connection reuse to cut full handshakes.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →