Engineering Atlas › Foundations

Processes, Threads and Async I/O

16 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Processes: a program with walls around it

When you start a program, the operating system creates a process. It gets a process ID, its own virtual address space (address 0x1000 in one process means nothing in another), a table of open file descriptors (files, sockets, pipes), and one thread to run main. The walls are the point: a bug that scribbles over memory in one process cannot touch another, and the OS can kill a process and reclaim everything it held.

Processes talk only through channels the OS provides: pipes, sockets, files, or explicitly shared memory. A web server that runs 8 worker processes behind one listening socket, as Nginx and Gunicorn do, gets crash isolation and uses 8 cores, but the workers cannot share an in-memory cache without extra machinery.

Threads: several workers in one house

A thread is a program counter, a set of registers and a stack. Every thread in a process sees the same heap, globals and open files. Creating one is cheaper than creating a process, and passing data between threads is just passing a pointer.

The danger is that sharing is the default. Consider count = count + 1 run by two threads. Each reads count (say 41), adds one, and writes back 42. One increment is lost. This is a race condition: the result depends on the timing of instructions you do not control. The fix is to make the read-modify-write happen as one step, with a mutex (only one thread inside the critical section at a time), an atomic instruction, or by giving the data a single owner that other threads send messages to.

Concurrency versus parallelism

Concurrency means several tasks are in progress at once, possibly taking turns on one core. Parallelism means several tasks are executing at the same instant on different cores. A chef with three pans on the stove is concurrent; three chefs are parallel. If your work is CPU-bound (image resizing, compression, model inference on CPU), you need parallelism, which means more cores and more threads or processes. If your work is I/O-bound (most web backends), you need concurrency: plenty of requests in flight while each one waits.

Blocking I/O and the thread-per-request model

The simplest server gives each request its own thread. The thread calls read on a socket, and if no data has arrived, the OS puts the thread to sleep until it does. This is blocking I/O, and the code reads top to bottom like a script. It works well up to a few hundred or a few thousand concurrent connections. Beyond that, each thread costs memory for its stack and the scheduler spends more time switching between threads than running them.

Async I/O and the event loop

The alternative is to mark sockets non-blocking and use one thread to watch many of them. The thread asks the kernel, "of these 10,000 sockets, which have data now?", handles the ready ones, and asks again. This is an event loop. Node.js, Python's asyncio, Nginx and Redis are built this way. In code it appears as callbacks, promises, or async/await: each await is a point where your function says "I am waiting; run something else and come back to me".

The event loop has one hard rule: never block it. A 200 ms synchronous call (a CPU-heavy hash, a blocking file read, a synchronous HTTP client) freezes every other connection on that loop for 200 ms. Runtimes push such work to a separate thread pool, and you should too.

Green threads: the best of both

Go takes a third path. Goroutines look like ordinary blocking threads in code, but they are managed by the Go runtime, not the OS. They start with a stack of a few KB that grows as needed (2 KB in early versions; since Go 1.19 the starting size follows the average stack use seen so far) (Go FAQ, Go 1.19 notes), and when one blocks on a socket, the runtime parks it and registers the socket with the same kernel readiness mechanism an event loop uses. You write simple sequential code; the runtime does the event loop for you. Java's virtual threads, final in Java 21, follow the same idea: the JDK schedules them onto a small set of carrier threads and unmounts them when they block on most I/O, and the JEP says never to pool them and to use a semaphore to limit concurrency instead (JEP 444). In Java 21 a virtual thread blocking inside synchronized stayed pinned to its carrier; Java 24 removed that limitation (JEP 491).

Down the stack

Layer 1The contract

The OS promises each process an isolated address space and its own file descriptors, and promises that runnable threads will each get CPU time eventually, in no particular order. It does not promise when a thread runs or for how long, and without synchronisation other threads may see shared memory updates late or reordered. A non-blocking read promises either some bytes or an immediate "would block" error (EAGAIN); it never sleeps.

Layer 2The mechanism

On Linux, a process and a thread are both a task_struct in the kernel. fork creates a task with a copy-on-write copy of the parent's memory; pthread_create calls clone with flags such as CLONE_VM and CLONE_FILES, so the new task shares memory and open files; CLONE_THREAD puts it in the same thread group, so all threads share one process ID (clone(2)). The scheduler keeps runnable tasks per core and gives each a slice of CPU time. Linux used CFS for years and began moving to EEVDF in 6.6, which also lets a task request a specific slice through sched_setattr() (kernel EEVDF docs).

The event loop is built on epoll: epoll_create1 makes an interest list, epoll_ctl adds sockets to it, and epoll_wait sleeps until at least one is ready, then returns only the ready ones (epoll(7)). Its cost tracks ready sockets, not watched ones. By default epoll is level-triggered, like poll but faster; edge-triggered mode (EPOLLET) reports only changes and requires non-blocking sockets read until EAGAIN. Node's libuv wraps epoll (kqueue on macOS) plus a default pool of 4 threads (UV_THREADPOOL_SIZE, maximum 1024) for work with no non-blocking form, such as file system calls and DNS lookups through getaddrinfo (libuv docs, Node.js guide). That small pool is itself a trap: four slow fs or dns.lookup calls leave everything else queued behind them. Go's runtime scheduler multiplexes goroutines (G) onto OS threads (M) through logical processors (P, one per GOMAXPROCS), and its netpoller uses epoll underneath.

Layer 3The machine

A context switch between threads saves and restores registers and switches stacks. One careful measurement on Linux found about 1.2 to 1.5 µs per switch when pinned to a core and about 2.2 µs unpinned (Bendersky). The hidden cost is larger: the new thread's data is not in the L1 and L2 caches, so it runs slowly for a while. Each OS thread reserves virtual memory for its stack: the default comes from the RLIMIT_STACK soft limit, typically 8 MB, and only touched pages use RAM (pthread_create(3)). In that same measurement, 10,000 threads showed about 80 GB virtual but only about 80 MB resident. So 10,000 threads is feasible but heavy, while 100,000 goroutines at a few KB each is routine.

When a packet arrives, the kernel appends its bytes to the socket's receive buffer and marks the socket ready, waking whichever thread sleeps in epoll_wait or read. An uncontended mutex is one atomic compare-and-swap, tens of nanoseconds. Under contention, the losing thread sleeps in the kernel via futex, which is entered only when blocking is likely, and the lock's cache line bounces between cores (futex(2)).

Why it's built this way

  1. Programs crash and misbehave, so the OS isolates them in processes with separate address spaces.
  2. Isolation makes sharing expensive, so threads exist to share memory cheaply inside one process.
  3. Servers spend most of their time waiting, and a sleeping OS thread per waiting connection wastes memory and scheduler time.
  4. So kernels added readiness APIs (select, then poll, then epoll and kqueue) that let one thread wait on many sockets at a cost tied to activity, not connection count.
  5. Callback-style code is hard to write, so languages added async/await and runtimes such as Go's hide the event loop behind ordinary-looking blocking code.

Diagram

sequenceDiagram
  participant L as Event loop thread
  participant K as Kernel (epoll)
  participant D as Database
  L->>K: epoll_ctl add sockets A, B, C
  L->>D: send query for A (non-blocking)
  L->>D: send query for B (non-blocking)
  L->>K: epoll_wait
  D-->>K: reply for B arrives
  K-->>L: B is readable
  L->>L: run handler for B, send response
  L->>K: epoll_wait
  D-->>K: reply for A arrives
  K-->>L: A is readable
  L->>L: run handler for A, send response

Worked example

An API endpoint spends 2 ms of CPU per request and 98 ms waiting on a database. Peak traffic is 2,000 requests per second.

Thread-per-request with a fixed pool: by Little's law (the JEP 444 text uses the same law to explain why thread-per-request caps throughput), requests in flight equal arrival rate times time in system, 2,000 × 0.1 s = 200, so the service needs at least 200 threads plus headroom. CPU needed is 2,000 × 2 ms = 4 core-seconds per second, so 4 cores busy. If the pool were capped at 100 threads, the service could only finish 100 / 0.1 s = 1,000 requests per second, and the rest would queue, even though the CPUs sit half idle. That is the classic sign of thread pool exhaustion: latency climbs while CPU stays low.

Event loop: one thread can hold all 200 in-flight requests because waiting costs only a registered socket. Its limit is CPU: 1 s / 2 ms = 500 requests per second per loop, so run 4 to 5 processes (one loop per core) behind a load balancer.

Now someone adds a synchronous password hash that takes 50 ms to one endpoint, called on the event loop. Each loop can now finish at most 20 of those per second, and every other request on that loop waits behind it. p99 latency jumps from about 100 ms to several seconds. The fix is to move the hash to a worker thread pool (Node's worker_threads, Python's run_in_executor) so the loop keeps serving.

In the wild

Trade-offs

ModelStrengthsWeaknessesChoose when
Multiple processesCrash isolation, uses all cores, no shared-memory racesHigher memory, coordination via IPCIsolation matters; runtimes with a global lock
OS threads, blocking I/OSimple sequential code, true parallelismMemory per thread, switch cost, data racesHundreds to low thousands of connections, CPU work
Event loop, async I/OTens of thousands of connections per threadOne blocking call stalls all, harder code pathsI/O-bound services, proxies, chat, gateways
Green threads (goroutines, virtual threads)Sequential code with event-loop efficiencyRuntime-specific, still need locks for shared dataHigh-concurrency services in Go or modern Java

Pitfalls in production

Leadership lens

Check yourself

  1. A service has low CPU usage but rising latency and a growing request queue. What is a likely cause?

    Show answer

    All worker threads are blocked waiting on a slow dependency (thread pool exhaustion), so new requests queue even though the CPUs are idle.

  2. Why can one event loop thread handle 10,000 connections while 10,000 OS threads struggle?

    Show answer

    A waiting connection on an event loop costs only a registered socket, and epoll_wait returns just the ready ones; each OS thread costs stack memory, kernel structures and context switches.

  3. Two threads each run x = x + 1 a million times on a shared x without a lock. Why can the result be less than two million?

    Show answer

    The increment is a separate read, add and write; if both threads read the same value before either writes, one update overwrites the other and is lost.

  4. When would you choose processes over threads for parallel work?

    Show answer

    When you need crash isolation, or the runtime cannot run threads in parallel (such as CPython with its global interpreter lock) for CPU-bound work.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →