Engineering Atlas › System Design Fundamentals

Running code without servers

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Execution environments

When an event arrives and no idle environment exists, the platform creates one: allocate a micro-VM, download the code package, start the runtime, and run your initialisation code. This is the cold start, typically 100 to 500 ms for small Node or Python functions and 1 to several seconds for large JVM packages. AWS says cold starts typically affect under 1% of invocations, more often in rarely called dev and test functions, and range from under 100 ms to over a second (Lambda docs). Afterwards the environment handles one request at a time and stays warm for an unspecified idle period, often minutes, though Lambda still recycles environments every few hours even under constant traffic. A second concurrent request gets a second environment, so concurrency equals the number of environments.

Scaling and limits

Concurrency is roughly arrival rate times average duration. At 500 requests per second and 200 ms average duration you need about 100 concurrent environments. Accounts and functions have concurrency quotas (1,000 per Region by default on Lambda), and scale-up is rate limited: each function can add at most 1,000 execution environments every 10 seconds, and requests beyond that get a 429 (Lambda scaling docs). A sudden 50x burst can therefore be throttled. Functions also have maximum durations (15 minutes on Lambda), memory caps, and payload size limits, which rule out long-running or stateful work.

Triggers and state

Functions are invoked by HTTP gateways, queue messages, object-store events, stream records, or schedules. Each trigger has its own retry behaviour: an async invocation that errors is retried twice, one and then two minutes apart, and a throttled one is retried for up to 6 hours; a queue redelivers until the message is deleted or sent to a dead-letter queue. AWS also warns that the async queue can deliver an event twice even when nothing failed (Lambda docs). That means handlers must be idempotent. Because environments are disposable, anything that must survive goes to a database, cache, or object store. Connection-heavy databases are a classic mismatch: 1,000 concurrent functions each opening a Postgres connection will exhaust the server, so use a connection proxy or an HTTP-native datastore.

Pricing model

You pay per invocation plus GB-seconds of execution: on Lambda in us-east-1, $0.20 per million requests and about $0.0000167 per GB-second on x86 (AWS Lambda pricing). Since 1 August 2025 the Init phase of on-demand ZIP functions on managed runtimes is billed too, so a slow cold start now costs money as well as latency (AWS Compute Blog). This is cheap for bursty traffic and expensive for constant high load, where a container fleet at 60% utilisation usually wins.

The line between serverless and servers is blurring. Lambda Managed Instances run functions on EC2 instances in your account at EC2 prices plus a 15% management fee; one environment serves many requests at once, and the fleet scales to a configured minimum instead of to zero (Lambda docs). Lambda durable functions checkpoint progress so a workflow can run for up to a year, well past the 15-minute limit of one invocation.

Down the stack

Layer 1The contract

The platform promises to run your handler for each event, eventually, with bounded resources. On AWS Lambda that means a maximum of 15 minutes per invocation, a configured memory size from 128 MB to 10,240 MB, request and response payloads up to 6 MB for synchronous calls (200 MB for a streamed response) and 1 MB for asynchronous ones, and one request at a time per execution environment on the default compute type (Lambda quotas). It does not promise that an environment will be reused, that anything in memory or /tmp survives, or that an asynchronous event runs exactly once: async invocations are retried twice on error, then sent to a dead-letter queue or failure destination.

Layer 2The mechanism

Each execution environment is a Firecracker microVM. Inside it, the language runtime runs a loop against the Runtime API: GET /2018-06-01/runtime/invocation/next blocks until an event arrives, the handler runs, and the runtime posts the result to /runtime/invocation/{request-id}/response. Before the first invoke, the Init phase loads the runtime and your code and runs initialisation outside the handler, limited to 10 seconds. Asynchronous events go into an internal queue first, so the caller gets a 202 immediately while a poller invokes the function. SnapStart takes a Firecracker snapshot of an initialised environment when you publish a version and restores it on demand, skipping most of Init. It now covers Java 11+, Python 3.12+ and .NET 8+, and code that generates unique IDs or random seeds during Init must move them after restore, because every environment starts from the same snapshot (SnapStart docs).

Layer 3The machine

Firecracker boots a minimal microVM in under 125 ms with less than 5 MiB of memory overhead per VM, and a host can create up to 150 microVMs per second (Firecracker), which is what makes dense multi-tenant packing affordable. A cold start adds downloading and unpacking your package, starting the runtime, and running init code. For a JVM that means class loading and interpreting code before the JIT warms up, which is why Java cold starts reach seconds. CPU is allocated in proportion to memory: at 1,769 MB a function gets the equivalent of one vCPU, so a 128 MB function also gets a small CPU slice and runs slower. Between invocations the environment is frozen, so background threads stop and timers drift; work started after returning a response may never finish.

Why it's built this way

  1. Customers' code is untrusted and shares hardware, so isolation must be VM-strength, not just containers.
  2. Full VMs boot too slowly and waste memory, so a stripped-down microVM monitor trades features for speed and density.
  3. Even a fast boot plus runtime start is too slow for every request, so environments are kept warm and reused.
  4. Billing is per millisecond of memory-time, so CPU is tied to memory to keep pricing one-dimensional.
  5. Reclaiming idle environments at will is what makes scale-to-zero pricing possible, so handlers must be stateless.

Diagram

sequenceDiagram
    participant C as Client
    participant G as HTTP gateway
    participant P as FaaS platform
    participant E as Execution env
    participant D as Managed DB
    C->>G: POST /resize
    G->>P: invoke function
    alt no warm environment
        P->>E: create env, load code, run init
        Note over P,E: cold start 100 ms to seconds
    else warm environment idle
        P->>E: reuse env
    end
    E->>D: read/write state
    D-->>E: ok
    E-->>P: response
    P-->>G: response
    G-->>C: 200 OK

Worked example

An image-thumbnail service receives uploads to object storage. Traffic is 2 million images a month, heavily concentrated in business hours, with peaks of 150 uploads per second.

Each thumbnail function uses 1 GB of memory and runs 400 ms. Monthly compute is 2,000,000 x 0.4 s x 1 GB = 800,000 GB-seconds. At roughly $0.0000167 per GB-second that is about $13, plus about $0.40 for requests. Peak concurrency is 150 x 0.4 = 60 environments, well within default quotas. A container alternative sized for the peak would need several always-on instances costing hundreds of dollars a month, mostly idle overnight.

Now compare an internal API at a steady 3,000 requests per second, 50 ms each, 512 MB. That is about 7.8 billion requests a month. Request charges alone are around $1,500 and compute around $3,200. A small autoscaled container fleet doing the same work might cost under $1,000. The break-even point depends on utilisation: serverless wins when average load is far below peak.

In the wild

Trade-offs

OptionStrengthsWeaknessesChoose when
FaaS functionsNo servers, scale to zero, per-request billingCold starts, duration and memory limits, vendor APIsSpiky, event-driven, or glue workloads
Serverless containersBring any container, scale to zero, longer requestsStill cold starts, less control than a clusterYou need custom runtimes but want managed scaling
FaaS on managed instancesFunction model at EC2 prices, many requests per environmentNo scale to zero, thread safety now your problemSteady, high-volume functions you do not want to rewrite
Container clusterFull control, predictable latency, cheap at steady loadYou run capacity planning, patching, and scalingHigh steady traffic or long-lived connections

Pitfalls in production

Leadership lens

Check yourself

  1. A function averages 300 ms and receives 2,000 requests per second. Roughly how many concurrent environments are needed?

    Show answer

    About 600, because concurrency is arrival rate times duration (2,000 x 0.3).

  2. Why are serverless handlers expected to be idempotent?

    Show answer

    Triggers deliver at least once and retry on errors or timeouts, so the same event can run the handler more than once.

  3. When does a container fleet beat FaaS on cost?

    Show answer

    At high, steady utilisation, where per-request and per-GB-second pricing exceeds the cost of always-on instances that stay busy.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →