Engineering Atlas › System Design Fundamentals

gRPC for service-to-service calls

12 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

The wire

A gRPC call is an HTTP/2 request: POST /package.Service/Method with content-type: application/grpc, headers for metadata (auth tokens, trace ids) and grpc-timeout for the deadline. The body is a sequence of length-prefixed messages: 1 byte compression flag, 4 bytes big-endian length, then the Protobuf bytes. The response sends headers, zero or more messages, and then trailers with grpc-status and grpc-message; the spec requires the status in trailers even when the call succeeds (gRPC HTTP/2 protocol). Because status lives in trailers, a call can stream for an hour and still report an error at the end.

HTTP/2 multiplexes many concurrent streams on one TCP connection, with per-stream flow control. A single connection can carry thousands of in-flight calls, which removes connection setup cost but concentrates load.

Contracts and codegen

The .proto file is the contract. Code generation gives each language a typed stub, so a field rename is a compile error rather than a production incident. Protobuf's evolution rules apply: add fields with new numbers, never reuse or retype them.

Deadlines and cancellation

gRPC sets no deadline by default, so a call with none can wait forever (gRPC deadlines). Each call should carry an absolute deadline. A server that calls downstream services passes the remaining time: the library turns the deadline into a relative timeout with the elapsed time already taken off, which also sidesteps clock skew between machines. Java and Go propagate it automatically; C++ needs you to turn it on. That way an edge request with a 300 ms budget never triggers a backend call that will run for 2 seconds. When a client cancels or the deadline passes, cancellation propagates down the call tree, freeing work that nobody will use. Without deadlines, slow dependencies accumulate stuck requests until threads or memory run out.

Status codes and retries

gRPC has 17 codes of its own, among them UNAVAILABLE (usually transient and worth retrying with backoff, though not always safe for a non-idempotent call) (gRPC status codes), DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED (rate limited), INVALID_ARGUMENT, NOT_FOUND, ABORTED (concurrency conflict), INTERNAL. A service config can declare retry policies per method, with maximum attempts, backoff and retryable codes, plus hedging (sending a second copy after a delay). Each backoff gets ±20% jitter, and a retryThrottling setting acts as a retry budget: each failure spends a token, each success earns back a fraction of one, and retries stop while the bucket is below half full (gRPC retry guide).

Load balancing

An L4 load balancer balances connections. With HTTP/2 a client opens one connection and sends everything over it, so all its traffic hits one backend, and new backends get nothing until clients reconnect. Fixes: an L7 proxy (Envoy, a service mesh) that balances per request, or client-side balancing where the client resolves all backend addresses and spreads calls with round robin or least-request. Servers should also set a maximum connection age: past it, the server sends GOAWAY and lets in-flight calls finish, with ±10% jitter so clients do not all reconnect at once (gRFC A9). A long-lived stream cannot be rebalanced once it has started, so streaming only where it pays is also a balancing decision (gRPC performance guide).

Browsers

Browsers cannot read HTTP/2 trailers through fetch, so they use gRPC-Web, which moves the trailers into the response body as a final message marked by the top bit of the flag byte. A proxy such as Envoy usually translates between the two (gRPC-Web protocol). The other route is a transcoding gateway that exposes the API as JSON.

Down the stack

Layer 1The contract

Every gRPC call ends with exactly one status: a code and an optional message. Messages within one stream arrive in order and intact. A call that outlives its deadline fails with DEADLINE_EXCEEDED on the client, and the server sees its context cancelled. Without a retry policy a call is attempted at most once from the application's view. The library may still retry transparently when it knows the server application never saw the request: as often as needed if the call never left the client, and once if it reached the server's gRPC library but not the handler (gRPC retry guide).

Layer 2The mechanism

On HTTP/2 a call is one stream. The client sends a HEADERS frame with :method POST, :path /pkg.Service/Method, te: trailers, content-type: application/grpc, grpc-timeout and optional grpc-encoding, then DATA frames holding the 5-byte-prefixed messages; one message can span several DATA frames or several messages can share one. The server answers with HEADERS, DATA, and a final HEADERS frame flagged END_STREAM that carries grpc-status. Cancellation is an RST_STREAM frame with error code CANCEL. Flow control uses WINDOW_UPDATE frames from an initial 65,535-byte window (RFC 9113, which replaced RFC 7540 in 2022), which gRPC grows by probing the bandwidth-delay product. Keepalives are PING frames, and a server that wants a client to reconnect elsewhere sends GOAWAY. On the client, a name resolver produces addresses and a load-balancing policy (pick_first, round_robin, or xDS-driven) picks a subchannel per call.

Layer 3The machine

Each HTTP/2 frame has a 9-byte header and HPACK shrinks repeated headers to a few bytes, so a small unary call costs well under 100 bytes of framing. All streams share one TCP connection, so a single lost segment stalls every stream behind it until the retransmit; on Linux the minimum retransmission timeout is 200 ms, which shows up as p99 spikes on lossy links. One busy connection is also served by one socket and often one I/O thread, which caps throughput and is another reason to keep a small pool of connections per backend. Encoding a 1 KB Protobuf message costs on the order of a microsecond. Cloud load balancers and NATs drop idle TCP flows after a timeout, which is why keepalive intervals must be shorter. AWS NLB defaults to 350 seconds; since 2024 TCP listeners can set 60 to 6,000 seconds, while TLS listeners stay fixed at 350 (AWS NLB docs).

Why it's built this way

  1. Internal services make many small calls, and new TCP and TLS handshakes are expensive, so gRPC multiplexes calls over long-lived HTTP/2 connections.
  2. Streaming responses only know their outcome at the end, so status goes in trailers rather than the HTTP status line.
  3. Many languages must agree on bytes and types, so the contract is an IDL with generated code.
  4. Deep call chains amplify slowness, so deadlines travel with the call and bound total work.
  5. Long-lived multiplexed connections make connection-level balancing meaningless, so balancing moves into the client or an L7 proxy.

Diagram

sequenceDiagram
    participant C as "Client stub"
    participant E as "Envoy (L7)"
    participant A as "Service A"
    participant B as "Service B"
    C->>E: HEADERS :path=/orders.Orders/Get, grpc-timeout=300m
    C->>E: DATA [0][len=42][protobuf]
    E->>A: same stream, picked per request
    A->>B: GetPrice, grpc-timeout=250m
    B-->>A: DATA + trailers grpc-status=0
    A-->>E: DATA [protobuf]
    A-->>E: TRAILERS grpc-status=0
    E-->>C: response stream

Worked example

A checkout service calls inventory, pricing and fraud services, 15,000 calls per second each, with p99 budgets that must fit inside a 400 ms user-facing deadline.

Switching inventory from JSON over HTTP/1.1 to gRPC: payload drops from 1.4 KB to 380 bytes, and serialisation CPU drops by about 4x. With HTTP/1.1, the client kept 600 connections open to avoid head-of-line blocking; with HTTP/2 it keeps 4 per backend host.

Then a problem: the inventory fleet scales from 20 to 30 pods, but the 10 new pods get almost no traffic, because checkout's existing connections still point at the first 20. Setting MAX_CONNECTION_AGE to 5 minutes on servers and using client-side round robin over DNS-discovered addresses spreads load within minutes.

Deadlines: checkout sets 400 ms. It spends 20 ms locally, then calls pricing with the remaining 380 ms. Pricing hits a slow cache and would take 2 seconds, but cancels at 380 ms, returning DEADLINE_EXCEEDED. Without propagation, pricing would have kept working for 1.6 seconds on 15,000 requests per second, 24,000 wasted in-flight requests during the incident.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
gRPCTyped contracts, compact, streaming, deadlinesNeeds L7 or client-side LB; not browser-nativeInternal service-to-service traffic
REST/JSONUniversal, cacheable, easy to debugLarger payloads, weak contractsPublic and browser-facing APIs
GraphQLClient-shaped queriesServer complexity, cachingAggregation for many UI clients
Message queueDecoupled, bufferedAsync, no direct responseWork that need not finish in the request

Pitfalls in production

Leadership lens

Check yourself

  1. Why does an L4 load balancer distribute gRPC poorly?

    Show answer

    It balances TCP connections, and gRPC multiplexes all calls on a few long-lived connections, so load sticks to whichever backends those connections reached.

  2. Why does gRPC put status in trailers rather than the HTTP status code?

    Show answer

    For streaming calls the outcome is known only at the end, after messages have already been sent.

  3. What does deadline propagation save during an incident?

    Show answer

    Work that no caller is waiting for; downstream calls stop when the original budget expires instead of piling up.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →