Engineering Atlas › System Design Fundamentals

How services find each other

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why a config file stops working

A config file of addresses works for three fixed servers. With autoscaling, rolling deploys and containers that get a new IP every restart, the list changes many times a day. This is the finding things fast problem shape combined with partial failure: the answer has to be fresh, cheap to read millions of times a second, and correct enough when some instances are half-dead. The same shape appears in a second domain: a ride-hailing app finding nearby available drivers is service discovery with geography, where drivers register, heartbeat their availability, and expire if the app goes silent.

Registration

In self-registration the instance calls the registry on startup ("I am payments, at 10.2.3.4:8443, zone b") and renews periodically; Netflix's Eureka clients work this way. In third-party registration something else watches the platform and registers on the instance's behalf: in Kubernetes, the EndpointSlice controller lists every pod that matches a Service's selector and passes its readiness probe. Third-party registration cannot be forgotten by a buggy instance, but it only knows what the platform knows.

Health and expiry

A registry needs to remove instances that crash without deregistering. The common tool is a lease with a TTL: the entry lives until now + ttl and each heartbeat pushes the expiry forward. Eureka clients renew every 30 seconds and the server evicts an instance after 90 seconds without a renewal (Netflix Eureka wiki). Consul runs an agent on every node, which runs health checks locally (HTTP, TCP, script, or a TTL check the app must refresh) and reports a status of passing, warning or critical; liveness of the agents themselves is tracked with gossip. A health status lets an instance be listed but excluded from traffic, for example during warm-up or graceful shutdown.

Lookup: server-side or client-side

Server-side discovery: the caller sends to a stable name or virtual IP, and something in the middle picks an instance, so callers stay simple. A Kubernetes ClusterIP Service works this way: kube-proxy programs iptables or nftables rules on every node that rewrite the VIP to a pod IP. If you learned this with IPVS mode, note that it has been deprecated since Kubernetes 1.35, will be off by default from 1.40 and removed in 1.43; iptables is still the default, and the docs say nftables will become the default in a future release (Kubernetes docs). Cloud load balancers with target groups are the same idea.

Client-side discovery: the caller fetches the instance list and chooses itself, using its own balancing algorithm, retries and zone preference. gRPC's name resolvers, Eureka with a client-side balancer, and Envoy (fed by the xDS endpoint discovery service) work this way. Netflix put its original balancer, Ribbon, into maintenance mode (Ribbon README); Spring applications now use Spring Cloud LoadBalancer, which defaults to round robin (Spring Cloud docs). There is no extra hop, at the price of a discovery library in every language, or a sidecar next to every process, which is what a service mesh is.

DNS sits in between. payments.prod.svc.cluster.local in Kubernetes, or Consul's DNS interface, returns instance IPs; SRV records (RFC 2782) add port, priority and weight. DNS is universal, but caches and TTLs make it slow to react, and most clients just take the first address.

Down the stack

Layer 1The contract

A registry promises that a registered instance stays listed until it deregisters, fails its health check, or misses renewals for one full TTL. It does not promise that a listed instance is alive; it promises that a dead one is removed within a bounded time: roughly TTL plus the caller's cache refresh interval. With Eureka's defaults that worst case is 90 seconds of lease, plus up to 30 seconds of server response cache, plus 30 seconds of client cache refresh: about two and a half minutes. Netflix's own docs warn that a change can take up to 2 minutes to reach all clients (Netflix Eureka wiki). Lookups are eventually consistent and may differ between callers. Registries built on consensus (Consul servers on Raft, etcd behind Kubernetes) give a linearizable write path, while reads are often served from caches or followers for scale.

Layer 2The mechanism

etcd models leases directly: LeaseGrant(TTL) returns a lease ID, keys written with that lease are deleted when it expires, and LeaseKeepAlive is a long-lived gRPC stream that renews it. Watchers on a key prefix get a stream of put and delete events, so clients learn about changes without polling. Consul gives the same effect through blocking queries: the client sends ?index=N, the server holds the request open (5 minutes by default, 10 at most, plus up to wait/16 of random jitter so clients do not all wake together) until the catalog's X-Consul-Index passes N, then returns the new list (Consul docs). Eureka clients fetch deltas every 30 seconds. Kubernetes splits each Service's endpoints into EndpointSlice objects of up to 100 endpoints by default (configurable up to 1,000), so one pod change rewrites a small object instead of the whole list; each endpoint carries ready, serving and terminating conditions, and its zone (Kubernetes docs). Envoy receives endpoints as ClusterLoadAssignment messages over EDS, grouped by locality with weights, which is how it prefers same-zone endpoints.

Layer 3The machine

Heartbeats are writes. 20,000 instances renewing every 10 seconds is 2,000 writes per second; on a Raft-backed store each write is an fsynced log append replicated to a quorum, which is why Consul does health checking on local agents and only sends changes to the servers, and why etcd leases are renewed by one keep-alive stream rather than per-key writes. Fan-out is the larger cost. Before EndpointSlices, a Service with 5,000 pods had a single Endpoints object of about 1.5 MB, and every pod change sent it to every node's kube-proxy; on 3,000 nodes, that is about 4.5 GB per change. Caches at every layer trade freshness for this cost: OpenJDK caches successful DNS lookups for 30 seconds unless networkaddress.cache.ttl says otherwise (OpenJDK source), glibc does not cache at all without nscd or systemd-resolved, and each client library holds its own list. A long-lived HTTP/2 or gRPC connection also hides changes: the client keeps using the endpoint it connected to until the connection breaks or hits a max-age.

Why it's built this way

  1. Instances crash without notice, so entries must expire on their own, which means leases and heartbeats.
  2. Heartbeats from a large fleet are a heavy write load, so health checking moves close to the instance (local agents, platform controllers) and only changes travel upward.
  3. Lookups vastly outnumber changes, so callers cache lists and subscribe to changes instead of querying per request.
  4. Caching makes every view stale, so callers must still handle dead endpoints with timeouts, retries and outlier ejection.
  5. A registry outage would stop all traffic, so clients keep using their last known list when the registry is unreachable.

Diagram

sequenceDiagram
  participant I as Instance B
  participant R as Registry
  participant C as Caller
  I->>R: register payments B zone-b (ttl 30s)
  loop every 10s
    I->>R: heartbeat B
  end
  C->>R: watch payments
  R-->>C: [A, B, C] passing
  Note over C: prefer same zone, round robin
  I-xR: crash, heartbeats stop
  Note over R: lease expires after 30s
  R-->>C: update [A, C]

Worked example

A company runs 400 services and 12,000 instances across three zones on Consul. Each instance has an HTTP health check every 10 seconds, run by the local agent, so the servers see only status changes: a few hundred per minute during deploys. Callers use blocking queries, so a change reaches them in well under a second. A payments instance that crashes outright is caught by gossip within a few seconds, because its node agent stops responding; one that hangs while its agent stays up is caught by the HTTP check after its 10-second interval and timeout, and removed from the next blocking query response.

Now compare a DNS-only design with a 60-second TTL. A deploy replaces all 30 payments instances over 5 minutes. Callers that cached the old addresses keep sending to drained instances for up to a minute after each replacement, and JVM callers that cache longer stay wrong even after that. With roughly 4,000 requests per second to payments and 1/30 of them going to a stale address at any moment during the deploy, that is over 100 failed requests per second unless callers retry. The fix is either client-side discovery with watches, or a stable VIP whose proxy tier updates in seconds.

Zone preference matters for cost and latency. If checkout sends 2 Gbps to payments spread evenly across three zones, two thirds of it, about 430 TB a month, crosses zones; at a typical $0.01 per GB in each direction that is roughly $8,600 a month that zone-aware routing mostly removes. Always fall back to other zones when the local one has no healthy instances.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
Static configNothing to runWrong after every changeA handful of fixed hosts
DNS (A or SRV records)Universal, every language supports itTTL caching, slow reaction, weak balancingCoarse discovery, legacy clients, across regions
Server-side (VIP or load balancer)Simple callers, central policyExtra hop, proxy tier to runPolyglot callers, external clients
Client-side libraryNo extra hop, smart balancing and zone awarenessLibrary per language, harder upgradesUniform stack, latency-sensitive internal calls
Sidecar or mesh (xDS)Client-side benefits without app changesProxy per pod, control plane to runMany services in many languages

Pitfalls in production

Leadership lens

Check yourself

  1. Why do registries use leases instead of asking instances to deregister when they stop?

    Show answer

    Crashed, frozen or partitioned instances cannot deregister. A lease expires on its own unless renewed, so the registry converges without the instance's cooperation.

  2. A caller uses DNS with a 30-second TTL, yet keeps hitting a removed instance for 10 minutes. Name two likely causes.

    Show answer

    A runtime or library caching DNS longer than the TTL, or a long-lived connection (HTTP/2, gRPC, a pool) opened before the change and never re-resolved.

  3. What does Eureka's self-preservation mode protect against, and what does it cost?

    Show answer

    It stops mass eviction when many renewals go missing at once, which usually signals a network problem, not mass death. The cost is that genuinely dead instances stay listed until renewals recover, so callers must handle failures.

  4. When is server-side discovery a better choice than client-side?

    Show answer

    When callers are written in many languages or are outside your control, or when you want policy changed centrally without client upgrades. You pay an extra hop and a proxy tier to operate.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →