Engineering Atlas › System Design Fundamentals

Managing service traffic with a service mesh

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Data plane

In the classic sidecar model, every pod gets a proxy (commonly Envoy, or Linkerd's Rust proxy). Iptables rules redirect the pod's inbound and outbound traffic through it. When service A calls service B, A's proxy resolves B's healthy endpoints, picks one with a load-balancing algorithm (round robin, least requests, or a peak-EWMA latency-aware choice), opens an mTLS connection to B's proxy, applies timeouts and retries, and records metrics and trace spans. B's proxy terminates mTLS, checks authorisation policy, and forwards to B over localhost.

Newer sidecarless designs (Istio ambient mode, eBPF-based meshes) move layer 4 work (mTLS, basic telemetry) into a per-node component and run layer 7 proxies only where needed, cutting per-pod overhead. In Istio's ambient mode, generally available since 1.24 in November 2024 (Istio), a Rust per-node proxy called ztunnel handles mTLS and L4 policy and tunnels traffic over HBONE, an HTTP CONNECT-based protocol; optional per-namespace Envoy "waypoint" proxies add L7 routing and policy (Istio docs). Istio's benchmark at 1,000 requests per second measures ztunnel at about 0.06 vCPU and 12 MB against 0.20 vCPU and 60 MB for a sidecar Envoy (Istio performance).

Control plane

The control plane watches the cluster for services and endpoints, compiles routing and policy into proxy configuration, and streams it to every proxy (Envoy's xDS APIs). It also runs a certificate authority that issues short-lived workload certificates, often valid for hours, tied to a workload identity such as a Kubernetes service account. The identity is written as a SPIFFE ID, a URI of the form spiffe://<trust domain>/<workload path> (SPIFFE). Proxies rotate certificates automatically, which is what makes mesh-wide mTLS practical.

What you get

What it does not do

The mesh cannot make a non-idempotent call safe to retry, cannot propagate trace context unless the application forwards the headers, and does not replace application-level authorisation on data.

Down the stack

Layer 1The contract

The mesh promises that traffic between workloads is encrypted and mutually authenticated with an identity tied to the workload, not its IP; that authorisation policy is enforced on every request; that configuration changes reach all proxies eventually; and that if the control plane is unavailable, proxies keep serving with their last accepted configuration and certificates until those expire. Envoy's xDS protocol states this directly: the last known configuration persists until the connection to the management server is re-established (Envoy xDS protocol).

Layer 2The mechanism

Envoy-based meshes use the xDS APIs over long-lived gRPC streams: LDS for listeners, RDS for routes, CDS for clusters, EDS for endpoints, and SDS for secrets, often multiplexed on one ADS stream. Each response carries a version_info and nonce; the proxy ACKs by echoing the new version, or NACKs by sending back the previous version plus an error_detail, so a bad config is rejected without replacing the good one. Workload identity follows SPIFFE: the control plane's CA signs a short-lived X.509 certificate whose URI SAN is a SPIFFE ID like spiffe://cluster.local/ns/payments/sa/payments-api, delivered over SDS by the local Istio agent and rotated before expiry (the agent requests 24-hour certificates by default, SECRET_TTL=24h; Istio docs). Authorisation policies match on that peer identity plus method and path.

Layer 3The machine

A TLS 1.3 handshake costs one round trip and, on each side, an ECDHE key exchange and an ECDSA signature or verification, tens to low hundreds of microseconds of CPU. Bulk encryption with AES-GCM runs at several GB per second per core on CPUs with AES-NI, so encrypting the data itself is cheap; handshakes are the expensive part, and pooled connections between proxies amortise them. Proxy memory scales with the config it holds, roughly clusters times endpoints, so a full mesh view of thousands of endpoints in every proxy wastes gigabytes fleet-wide unless scoped. Each endpoint change triggers EDS pushes to every interested proxy; a rolling deploy of a large service can generate thousands of pushes per second, so control planes debounce and batch.

Why it's built this way

  1. Pod IPs change constantly, so identity is bound to a service account through certificates rather than to addresses.
  2. Long-lived keys are dangerous if leaked, so certificates are short-lived and rotated automatically, which requires automation.
  3. Thousands of proxies need consistent config without restarts, so it is streamed with versions and ACK or NACK.
  4. Handshakes are costly while bulk encryption is cheap, so proxies keep connections open and reuse them.

Diagram

flowchart TB
    CP["Control plane (config, discovery, CA)"]
    subgraph PODA["Pod: checkout"]
        A["checkout app"] --> PA["Proxy"]
    end
    subgraph PODB["Pod: payments"]
        PB["Proxy"] --> B["payments app"]
    end
    PA -->|"mTLS, retries, LB"| PB
    CP -.->|"xDS config + certs"| PA
    CP -.->|"xDS config + certs"| PB
    PA -.->|"metrics, spans"| OBS["Telemetry backend"]
    PB -.->|"metrics, spans"| OBS

Worked example

A fintech runs 220 services on Kubernetes across 2,500 pods, in Go, Java, and Python. Auditors require encryption in transit and proof of which services can reach the card vault. Today, each language has its own TLS setup, and retry behaviour differs by team.

They adopt a sidecar mesh. Measured overhead per proxy: about 40 MB memory and 0.1 vCPU at their median load, so roughly 100 GB of memory and 250 vCPU fleet-wide, about 7% of cluster cost. Each hop adds two proxy traversals: 0.6 ms at p50 and 2.5 ms at p99. A checkout request crossing 6 services gains about 4 ms at p50.

In return, mTLS reaches 100% of traffic within two months, authorisation policy restricts the card vault to 3 caller identities, and every service pair has latency and error dashboards. A canary of a new pricing service at 5% weighted traffic catches a 2% error regression, rolled back by changing one routing resource.

Two incidents follow. A control plane upgrade pushes a malformed config that breaks routing for 6 minutes, so they add config validation and a staged rollout per namespace. And a mesh-level retry policy of 3 attempts amplifies load on a struggling service; they switch to retry budgets capping retries at 20% of requests.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Libraries in each serviceLowest latency, no extra infrastructurePer-language implementations, upgrades need redeploysFew languages, small number of services
Sidecar meshUniform policy and telemetry, per-workload controlPer-pod resource cost, extra hops, operational complexityLarge polyglot fleets with security requirements
Sidecarless or ambient meshLower overhead, simpler pod lifecycle, GA in Istio since 1.24L7 features need separately run waypoint proxies; shared per-node L4 proxyLarge fleets where sidecar cost dominates
No mesh, gateway onlySimple, controls only north-south trafficNo east-west security or visibilitySmall systems or a monolith with few services

Pitfalls in production

Leadership lens

Check yourself

  1. What is the difference between the data plane and the control plane?

    Show answer

    The data plane is the proxies that carry and act on every request; the control plane computes configuration, discovery, and certificates and distributes them to the proxies, without sitting on the request path.

  2. Why can a mesh not fully implement distributed tracing on its own?

    Show answer

    Proxies see separate inbound and outbound requests and cannot link them unless the application copies trace headers from the incoming request to its outgoing calls.

  3. Why is mesh-level retry configuration risky?

    Show answer

    Retries at the mesh multiply with application and client retries, so a struggling service can receive several times its normal load. Retry budgets and a single owning layer prevent this.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →