Engineering Atlas › Foundations

Linux and the Command Line for Debugging

14 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Processes, files and the shell

A running program is a process with a numeric process ID (PID), an owner, a current directory, environment variables and a list of open files. The shell (usually bash or zsh) is itself a program: it reads your command line, starts processes and waits for them; in a pipeline it waits for every command before it returns a status (bash(1)). Linux follows the idea that "everything is a file": regular files, directories, sockets, pipes and devices are all accessed through file descriptors, small integers that index the process's table of open files.

Streams, redirection and exit codes

Each process starts with three file descriptors: 0 is standard input, 1 is standard output, 2 is standard error. By default all three point at your terminal. Redirection changes that: cmd > out.txt sends output to a file, 2>&1 sends errors to the same place as output, and < in.txt reads input from a file. When a process finishes, it returns an exit code: 0 means success, anything else means failure. echo $? shows the last one. Scripts and CI pipelines depend on this, so a tool that prints an error but exits 0 is a bug.

Pipes

cmd1 | cmd2 connects the output of cmd1 to the input of cmd2. Both run at the same time. This lets you build answers from small tools, for example the five most common error paths in a log:

grep ' 500 ' access.log | awk '{print $3}' | sort | uniq -c | sort -rn | head -5

grep keeps matching lines, awk picks the third field, sort groups identical paths together, uniq -c counts each group, sort -rn orders by count, and head keeps the top five.

The debugging toolkit

QuestionCommands
Is it running?ps aux, pgrep -a name, systemctl status name
What did it log?tail -f file, less file, journalctl -u name --since "10 min ago", grep
Is the machine busy?top or htop, uptime, vmstat 1 (load average counts waiting tasks too, see below)
Is memory short?free -m, dmesg (look for the OOM killer)
Is the disk full?df -h, du -sh dir/*
What is listening or connected?ss -tlnp, ss -tan
What files and sockets does it hold?lsof -p PID, ls -l /proc/PID/fd
Can I reach the service?curl -v URL, dig name, nc -zv host port
What is it doing right now?strace -p PID, perf top

Signals and permissions

Signals are small notifications the kernel delivers to a process. kill PID sends SIGTERM, which asks the process to shut down cleanly; a well-written server finishes in-flight requests and exits. kill -9 PID sends SIGKILL, which cannot be caught: the process dies immediately, possibly mid-write. Every file has an owner, a group and read, write and execute permissions for each, shown by ls -l. "Permission denied" usually means the process runs as a different user than you expect.

A debugging routine

Start broad and narrow down. First, is the process up and what do its last log lines say? Second, check resources with the USE method: for every resource (CPU, memory, disk, network), look at utilisation, saturation (extra work queued that it cannot service) and errors (Gregg, USE Method). One trap is the load average: on Linux it counts tasks that are runnable and tasks in uninterruptible sleep (state D in ps, usually blocked on disk or network storage), so a load of 40 with 5% CPU points at I/O, not computation (Gregg, Linux Load Averages). Third, reproduce the symptom with curl from the same machine. Only then reach for heavy tools like strace. This order finds the full disk or the crashed process in two minutes, before you spend an hour reading code.

Down the stack

Layer 1The contract

A command is a program that receives a list of string arguments, an environment, and three open streams, and returns an 8-bit exit status. In a pipeline, all commands run concurrently, and the shell's exit status is the last command's unless set -o pipefail is on, in which case it is the last (rightmost) command that failed (bash(1)). Writes smaller than PIPE_BUF (4096 bytes on Linux) are atomic, so two writers cannot interleave inside one small write (pipe(7)). If a process is killed by signal N, the shell reports exit code 128 + N: 137 means SIGKILL (often the out-of-memory killer), 143 means SIGTERM. Container runtimes reserve a few codes of their own; Docker uses 125 when the daemon fails and 126 or 127 when the command cannot be run or found (Docker docs).

Layer 2The mechanism

To run grep x f | wc -l, the shell calls pipe() to get a read end and a write end, then fork() twice to create two child processes. In the first child it uses dup2() to make the pipe's write end become file descriptor 1, then calls execve() to replace itself with grep. The second child makes the read end its descriptor 0 and becomes wc. The shell closes its copies and calls waitpid() for both. Neither program knows it is in a pipeline; each just reads descriptor 0 and writes descriptor 1. /proc is a virtual file system: reading /proc/1234/status asks the kernel to generate text about process 1234 on the spot, so ps is just a program that reads /proc (proc(5)).

Layer 3The machine

Each read or write is a system call, a switch from user mode into the kernel that costs on the order of 100 ns to a few µs. A Linux pipe is a kernel buffer, 64 KB (16 pages) by default since Linux 2.6.11, though newer kernels hand out less when a user already holds many pipes; a program can resize it with F_SETPIPE_SZ, up to 1 MiB for unprivileged users by default (pipe(7)). When it is full, the writer sleeps until the reader drains it; when it is empty, the reader sleeps. If the reader exits early, as head does, the next write gets SIGPIPE and the writer stops (pipe(7)), which is why huge-command | head returns quickly. grep over a file reads blocks from the page cache, so a second search of the same log is far faster than the first. strace uses the ptrace interface to stop the target at every system call, which can slow a syscall-heavy process by more than 100 times in the worst case (Brendan Gregg measured a dd run going from 0.10 s to 46 s) (Gregg, strace). Use it briefly in production, or prefer perf trace, which buffers events and costs far less.

Why it's built this way

  1. Early Unix machines were small, so programs were small and did one job.
  2. Small programs are only useful if they compose, so they all speak plain text over standard streams.
  3. Composition needs concurrency without huge buffers, so pipes are small bounded buffers that block the faster side.
  4. Treating devices, sockets and process state as files means one set of tools (cat, grep, ls) inspects all of them.
  5. The result has lasted 50 years: the same commands work on a laptop, a container and a 500-node fleet.

Diagram

sequenceDiagram
  participant Sh as Shell
  participant K as Kernel
  participant G as grep
  participant W as wc
  Sh->>K: pipe() returns read and write ends
  Sh->>K: fork() then dup2 and execve grep
  Sh->>K: fork() then dup2 and execve wc
  G->>K: write(1, lines) into 64 KB pipe buffer
  K-->>G: blocks if buffer is full
  W->>K: read(0) drains buffer
  G->>K: exit(0)
  W->>K: read returns 0 bytes (end of file)
  W->>K: exit(0)
  Sh->>K: waitpid() collects both exit codes

Worked example

At 14:05, alerts show an order API returning HTTP 502 from its load balancer for about 30% of requests. You SSH into one of its three hosts.

systemctl status orders says the service is active, but it restarted at 14:02. journalctl -u orders --since 13:55 shows normal request logs, then "write /var/log/orders/app.log: no space left on device", then a crash and restart. df -h confirms: /var is at 100% of 50 GB.

du -sh /var/log/* shows /var/log/orders using only 3 GB. Where did 47 GB go? lsof +L1 selects open files that have been unlinked (lsof(8)), meaning deleted but still held open, and there it is: a 44 GB debug.log that someone deleted an hour ago while an old log shipper process still had it open. A deleted file's space is only freed when the last file descriptor to it closes. Restarting the log shipper frees 44 GB in a second.

Now the counts. grep ' 502 ' /var/log/nginx/access.log | wc -l gives 18,240 lines in the last half hour, and the pipeline from earlier shows 17,900 of them on POST /orders, the one endpoint that writes a log line per request. The other two hosts had more free space and were fine, which explains 30% rather than 100%.

The lasting fixes come from the debugging: disk-usage alerts at 80%, log rotation with size limits, and a runbook entry for lsof +L1. Total time to diagnosis was about 10 minutes, using six commands and no code changes.

In the wild

Trade-offs

ApproachStrengthsWeaknessesChoose when
Shell on the hostFast, always there, sees everythingManual, one machine at a time, risky with write accessLive incidents, one host misbehaving
Central logs and metricsFleet-wide, searchable, historyDelay, sampling, may miss the crash momentPatterns across many hosts, trends
strace and perfSee exact system calls and hot functionsOverhead, needs privileges, noisy outputA process is stuck or slow and logs say nothing
Shell pipelines over logsQuick ad hoc queries, no setupHard to repeat, fragile parsingOne-off questions during an investigation
Scripts in the repoRepeatable, reviewed, testableSlower to writeQuestions you ask every week

Pitfalls in production

Leadership lens

Check yourself

  1. A container stops and its runtime reports exit code 137. What most likely happened?

    Show answer

    137 is 128 + 9, so the process was killed with SIGKILL, very often by the kernel's out-of-memory killer, which picks a task to sacrifice when reclaim cannot free memory (kernel docs); check dmesg or the container runtime's events for an OOM message.

  2. Why does cat huge.log | head -1 finish immediately instead of reading the whole file?

    Show answer

    head exits after one line and closes the pipe, so cat's next write receives SIGPIPE and cat stops.

  3. You deleted a 20 GB file but df still shows the disk full. Why?

    Show answer

    A process still has the file open; its blocks are only released when the last file descriptor to it closes, so find it with lsof +L1 and restart or signal that process.

  4. Why does a pipe have a small fixed buffer instead of growing as needed?

    Show answer

    Blocking the writer when the buffer is full applies backpressure, so a fast producer cannot run a slow consumer out of memory; the two processes run at the speed of the slower one.

Sources and further study

Primary

Systems docs

Books

Other curricula

Watch

Free resources

Open in the interactive app: practice, quiz, mock interview →