Reusing workers with a thread pool

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

A thread pool keeps a fixed or bounded set of worker threads alive and feeds them tasks through a queue, so a program pays thread creation once and caps how many things run at the same time.
Its real job is admission control: it decides what happens when work arrives faster than it can be done. The answer is some mix of queueing, growing the pool, running the task on the caller, or rejecting it.
Size it from the work: about one thread per core for CPU-bound tasks, more for tasks that block on I/O (cores × (1 + wait time / compute time)), and always with a bounded queue.
The classic bugs are an unbounded queue that hides overload until memory runs out, a task that waits on another task in the same pool (starvation deadlock), and exceptions that disappear inside futures.
Java's ThreadPoolExecutor grows past its core size only after the queue is full, which surprises almost everyone. Read the admission rules of whatever pool you use before tuning it.

How it works

The problem

An image service resizes uploads. The first version starts a new thread per request. At 50 requests per second this works. During a marketing push traffic hits 2,000 per second, the process creates 2,000 threads, each with its own stack, the CPU spends its time switching between them, latency climbs for everyone, and finally thread creation fails with "unable to create native thread". The server did not run out of CPU; it ran out of a way to say "not now".

A thread pool fixes both halves. Threads are created once and reused, so the per-task cost of creating one (tens of microseconds plus stack memory) disappears. And the pool size is a hard limit on concurrency: with 16 workers, at most 16 resizes run at once, and everything else waits in a queue or is turned away.

The parts

Every pool has the same four pieces:

Workers: long-lived threads that loop forever, taking a task from the queue and running it.
Work queue: a thread-safe queue between submitters and workers. Workers block on it when it is empty.
Admission policy: what submit does when all workers are busy and the queue is full.
Lifecycle: how workers start, how idle extra workers are retired, and how shutdown drains or abandons queued work.

A worker's loop is short:

def worker_loop(queue):
    while True:
        task = queue.take()          # blocks while empty
        if task is SHUTDOWN:
            return
        try:
            task.run()
        except Exception as e:
            task.fail(e)             # never let one task kill the worker

The queue is the producer-consumer pattern (see Producer Consumer Pattern); the pool is that pattern plus a lifecycle and a policy for overload.

Java's admission rules, the canonical example

ThreadPoolExecutor takes a core size, a maximum size, a queue and a keep-alive time. On execute(task) it does exactly this:

If fewer than corePoolSize threads exist, start a new thread with this task, even if other threads are idle.
Otherwise, try to put the task in the queue. If it fits, done; an idle worker will take it.
If the queue is full and fewer than maximumPoolSize threads exist, start a new thread with this task.
Otherwise, call the RejectedExecutionHandler.

Two consequences matter. First, with an unbounded queue (which Executors.newFixedThreadPool uses), step 2 never fails, so the pool never grows beyond core and never rejects; overload turns into an ever-longer queue and rising memory. Second, with a bounded queue, a burst first fills the queue and only then adds threads, so the tasks that arrive last may run before tasks queued earlier. Threads above the core count that stay idle for keepAliveTime are retired.

The four built-in rejection handlers are the menu of overload policies: AbortPolicy throws, CallerRunsPolicy runs the task on the submitting thread (which slows the submitter down, a crude but effective form of backpressure), DiscardPolicy silently drops it, and DiscardOldestPolicy drops the oldest queued task and retries. Silently dropping is almost never what you want.

Sizing

For CPU-bound work, more threads than cores only adds context switches; use about the number of cores. For work that blocks, a thread that spends 90 ms waiting on a database for every 10 ms of CPU leaves the core free most of the time, so the rough rule is cores × (1 + wait / compute): 8 × (1 + 9) = 80 threads. Little's law gives the same answer from the outside: to sustain 800 requests per second at 100 ms each, you need 80 in flight. Then check the downstream: 80 threads that each hold a database connection need a connection pool of at least 80, and the database must be able to take it. The pool is usually the cheapest place to enforce the limit of the slowest dependency.

Separate pools for separate work

One pool for everything means a slow dependency can take every worker, and then fast, unrelated requests wait behind it. Giving each dependency or workload its own pool is the bulkhead pattern (see Bulkhead Pattern). Node.js does a version of this internally: libuv runs file system calls, dns.lookup, some crypto and zlib on a thread pool of 4 by default, separate from the event loop, and a burst of slow DNS lookups can delay unrelated file reads because they share those 4 threads.

Work stealing

A single shared queue becomes a point of contention when tasks are tiny. Work-stealing pools give each worker its own double-ended queue: a worker pushes and pops its own tasks at one end, and an idle worker steals from the other end of someone else's. Java's ForkJoinPool, Go's runtime scheduler and Tokio's multi-threaded runtime all work this way. It suits recursive, fine-grained tasks; for coarse request handling a shared queue is simpler and fair.

Reviewing agent-written code

Agents tend to create a new executor inside a function that is called per request (with ThreadPoolExecutor() as ex: in a handler), which recreates threads on every call and defeats the pool. They also call future.result() from inside a task running on the same pool, which deadlocks once all workers are doing it. Ask for one pool per workload, created at startup and injected, with a bounded queue, an explicit rejection policy, named threads, and every future's exception either awaited or logged.

The same shape elsewhere

A thread pool is a limited set of servers in front of a queue, the overload shape. A database connection pool, a Kubernetes cluster with a fixed number of nodes, a call centre with 20 agents and a hold queue, and an AWS Lambda concurrency limit all have the same three knobs (how many servers, how long a queue, what to do when both are full) and the same failure: an unbounded queue turns a brief overload into minutes of latency.

Down the stack

Layer 1The contract

A pool promises: every accepted task runs exactly once on some worker; no more than max tasks run concurrently; a rejected task is reported to the submitter (by exception, return value, or running it on the caller), never lost without a signal unless the policy says so; a task's exception is delivered to its future and does not kill the worker; after shutdown, no new tasks are accepted and queued ones either run or are returned. It does not promise ordering between tasks, and under Java's rules not even FIFO order of submission.

Layer 2The mechanism

ThreadPoolExecutor packs the run state and the worker count into one AtomicInteger called ctl (3 bits of state, 29 bits of count) so that "check count and add a worker" is one compare-and-set. Each Worker is a small lock around a thread; getTask() calls queue.poll(keepAliveTime) when the pool has more than core threads and queue.take() otherwise, which is how idle extra threads time out. Python's ThreadPoolExecutor uses an unbounded SimpleQueue and starts threads lazily up to max_workers, which defaults to min(32, CPU count + 4) in recent versions. Go has no pool in the standard library because goroutines are cheap; the idiom is a fixed number of goroutines ranging over a channel, or a semaphore (a buffered channel or errgroup.SetLimit) to cap concurrency. Go's scheduler itself is a work-stealing pool: each P has a local run queue of 256 goroutines plus a global queue, and an idle P steals half of another P's queue.

Layer 3The machine

On Linux, pthread_create is a clone() call with flags such as CLONE_VM | CLONE_FILES | CLONE_THREAD, plus an mmap for the stack: 8 MB of virtual address space by default for C threads, 1 MB for Java threads unless -Xss says otherwise. Physical memory is touched lazily, but each thread still costs kernel structures and tens of microseconds to create and join. A context switch between threads costs roughly 1 to 5 us directly, and more indirectly when the incoming thread finds the caches cold. When a worker blocks on an empty queue, the JVM parks it with a futex wait; a submit wakes one sleeper with futex wake, a syscall of about a microsecond. Hot pools spin briefly before parking to avoid that cost. Java 21 virtual threads change the arithmetic: they are scheduled by the JVM onto a small ForkJoinPool of carrier threads, start with a stack of a few hundred bytes, and make pooling threads unnecessary for blocking I/O, though you still need a semaphore to cap concurrency to a downstream.

Why it's built this way

Creating and destroying OS threads costs syscalls, memory and microseconds, so threads are created once and reused.
Each running thread competes for cores and caches, and each blocked one holds memory and often a connection, so concurrency must be capped.
Work arrives in bursts, so a queue absorbs short spikes instead of rejecting them.
A queue without a bound converts long overload into unbounded latency and memory, so the queue is bounded and the pool needs an explicit policy for the overflow.
One shared queue becomes contended with tiny tasks, so high-throughput runtimes split it per worker and steal.

Diagram

flowchart TD
  S["submit(task)"] --> A{"workers < core?"}
  A -- yes --> N1["start worker with task"]
  A -- no --> I{"idle worker?"}
  I -- yes --> R["idle worker runs task"]
  I -- no --> Q{"queue has room?"}
  Q -- yes --> E["enqueue"]
  Q -- no --> M{"workers < max?"}
  M -- yes --> N2["start extra worker with task"]
  M -- no --> P["rejection policy: abort, caller runs, drop oldest"]
  E --> W["worker finishes, takes next from queue"]

Worked example

A report service renders PDFs. Each render takes 40 ms of CPU and 160 ms waiting on two internal APIs. The hosts have 8 cores. Sizing: 8 × (1 + 160/40) = 40 threads. At 200 ms per task, 40 threads complete 200 renders per second.

The team first deployed Executors.newFixedThreadPool(40). During month-end, traffic rose to 260 per second for 15 minutes. The pool could not grow or reject, so the unbounded queue grew by 60 tasks per second, reaching 54,000 tasks. Queue wait passed 4 minutes, clients timed out after 30 seconds and retried, and the service kept rendering PDFs nobody was waiting for. Heap filled with queued request bodies and the JVM spent most of its time in GC.

The fix was a ThreadPoolExecutor(40, 40, ..., new ArrayBlockingQueue<>(400), new AbortPolicy()). A queue of 400 at 200 per second is two seconds of buffering, well inside the client timeout. Excess requests got an immediate 503 with Retry-After, the load balancer moved them to other hosts, and month-end latency stayed under 2.5 seconds. They also dropped tasks whose client deadline had already passed when a worker picked them up, which removed a quarter of the wasted work during the next spike.

Trade-offs

Option	Strengths	Weaknesses	Choose when
Thread per task	Simple; no queueing delay	Unbounded threads; collapses under load	Low, predictable task rates only
Fixed pool, unbounded queue	Never rejects; easy	Overload becomes memory and latency growth	Batch jobs where waiting is fine
Fixed pool, bounded queue, abort	Predictable latency; fast failure	Callers must handle rejection	Request serving behind a load balancer
Bounded queue, caller runs	Natural backpressure on producers	Slows the submitting thread, which may be an I/O loop	Internal pipelines where the producer can slow down
Work-stealing pool	Low contention for tiny tasks	No global ordering; harder to reason about	Recursive or fine-grained parallel computation
Virtual threads plus semaphore	Cheap blocking; no pool sizing for I/O	Still needs explicit caps for downstreams	Blocking I/O workloads on Java 21 or later

Pitfalls in production

Unbounded queues hiding overload until memory or timeouts explode. Fix: bound the queue to a few seconds of throughput and reject or shed beyond it.
Starvation deadlock: a task submits a subtask to the same pool and waits for it, and when every worker does that, none are left to run subtasks. Fix: separate pools per stage, or make the parent not block.
Exceptions lost in futures nobody reads. Fix: always await or attach a callback, and set an uncaught exception handler that logs with the thread name.
Thread-local state (user IDs, transactions, MDC logging context) leaking from one task to the next on a reused thread. Fix: clear it in a finally, or pass context explicitly.
One slow dependency occupying every worker. Fix: per-dependency pools or semaphores, and timeouts on every blocking call.
Executors created per request or never shut down, leaking threads. Fix: create pools at startup, inject them, and shut them down on exit.

Leadership lens

Ask for every pool: what is its size, its queue bound, its rejection policy, and which metric shows saturation (active threads, queue depth, rejections per second, queue wait time).
Treat pool size as a contract with downstreams; a pool of 80 against a database that allows 50 connections is a design bug, not a tuning problem.
Prefer failing fast at the edge over queueing deep inside; a 503 that a load balancer can retry elsewhere is cheaper than a 4-minute queue.
Standardise pool construction in a shared library with sane defaults (bounded, named, instrumented), so teams and coding agents do not reach for the unbounded convenience constructors.
Revisit the model when the platform changes: virtual threads, async runtimes and serverless concurrency limits each move where the limit should live.

Check yourself

A ThreadPoolExecutor has core 10, max 50 and an unbounded LinkedBlockingQueue. How many threads will it ever run, and why?
Show answer
10. Extra threads are only started when the queue rejects an offer, and an unbounded queue never does, so the max size is never reached.
Why is cores × (1 + wait/compute) a starting point and not the answer?
Show answer
It ignores downstream limits such as connection pools and the dependency's own capacity, and the wait and compute times change with load; measure queue wait and saturation and adjust.
How does CallerRunsPolicy create backpressure, and when is it dangerous?
Show answer
The submitting thread runs the task itself, so it cannot submit more until that finishes. It is dangerous when the submitter is an event loop or acceptor thread, because the whole server stalls while it runs one task.
A pool of 8 threads hangs with all workers in future.get(). What happened?
Show answer
Each task submitted a subtask to the same pool and blocked on it; with all 8 workers blocked, no worker is free to run the subtasks, so it is a starvation deadlock.

Watch

Java ExecutorService - Part 1 - Introduction (Defog Tech): how a fixed pool, its queue and its workers fit together, with sizing advice for CPU and I/O work.

Free resources

ThreadPoolExecutor (Java SE 21 docs): the exact admission rules, queuing strategies and rejection policies.
concurrent.futures (Python docs): Python's pool API, defaults for max_workers, and shutdown semantics.
Thread pool work scheduling (libuv docs): the small pool behind Node.js file system and DNS calls, and how to size it.
Making the Tokio scheduler 10x faster (Tokio blog): a detailed walk through a work-stealing pool's queues and wake-ups.
JEP 444: Virtual Threads (OpenJDK): why thread-per-task returns when threads become cheap, and why you should not pool virtual threads.
Go runtime scheduler source (proc.go): per-P run queues and work stealing as implemented.

Open in the interactive app: practice, quiz, mock interview →