Virtual Memory and Page Faults: What Actually Happens When RAM Runs Out

Your process asks the kernel for 8 GB. The kernel says yes. Neither of you has checked whether 8 GB of DRAM exists.

That gap — between the address space a process sees and the physical memory behind it — is where every out-of-memory incident lives. A container that dies with exit code 137, a build box that goes unresponsive for four minutes without ever crashing, a service whose p99 triples under no extra request load: same mechanism, three different angles.

Every memory access is a lookup, and the lookup can miss

When your code dereferences a pointer, the CPU hands a virtual address to the MMU, which walks the page tables to find the physical frame behind it. On x86-64 and arm64 the default page size is 4 KiB, with 2 MiB and 1 GiB huge pages available. Recently used translations live in the TLB, so the common case never touches the page tables at all.

When the page table entry says “not present,” the CPU raises a page fault and the kernel’s handler decides what kind it is:

A third outcome exists: no valid mapping at all, which becomes SIGSEGV.

The cost column is the entire story. A minor fault is bookkeeping. A major fault is a synchronous I/O the CPU stalls on, three to four orders of magnitude slower. Two processes can report identical fault counts and behave completely differently depending on the split.

Demand paging is why this matters at allocation time too. malloc(1 << 30) that you never write to costs you almost no physical memory — the kernel hands back address space and assigns frames on first touch. That is why RSS lags your allocations, why RSS jumps when you memset a buffer you already allocated, and why VSZ is close to useless as a capacity signal.

Reclaim, thrash, kill

As free pages fall below the kernel’s watermarks, reclaim starts. kswapd does it in the background; if an allocation can’t wait, the allocating process enters direct reclaim and stalls inside its own allocation call — invisible in application-level profiling, very visible in latency graphs.

Reclaim ranks candidates by how cheap they are to drop:

1. Clean file-backed pages. Free them immediately. If someone needs the data again, that’s a major fault later.
2. Dirty file-backed pages. Write back first, then free.
3. Anonymous pages (heap, stack, anything with no file behind it). These have nowhere to go except swap, zram, or zswap.

With swap disabled, step 3 is unavailable, so anonymous memory becomes effectively unevictable and all pressure lands on the page cache. The kernel starts evicting file-backed pages it needs immediately — including the executable text of running binaries — and faults them straight back in. That is thrashing: load average climbs, throughput collapses, the box stays technically alive, and ssh takes 40 seconds to echo a character. The tell is the major fault rate, not the free memory number.

When reclaim can’t free enough, the kernel OOM killer fires. It scores candidates roughly by memory footprint, adjusted by each process’s oom_score_adj (range -1000 to 1000, where -1000 makes a task ineligible). It is a last resort by design, which means by the time it acts you have usually already spent minutes in stall. Userspace killers like systemd-oomd and earlyoom exist precisely to act on PSI stall time instead of waiting for total allocation failure.

Containers change the boundary, not the mechanism. Under cgroup v2, exceeding memory.max triggers a cgroup-scoped OOM kill even when the host has free RAM to spare. The victim gets SIGKILL, the container exits 137 (128 + 9), and Kubernetes labels it OOMKilled. memory.high is the softer sibling: it throttles the cgroup and pushes it into reclaim rather than killing it.

One more knob explains why you rarely see malloc return NULL: vm.overcommit_memory defaults to 0, a heuristic that approves most requests. Set it to 2 with a strict overcommit_ratio and allocations start failing honestly at request time instead of turning into a kill later. Most people leave it at 0 and accept the trade.

Reading the actual signal

Before changing anything, find out which of the three stages you’re in.

• cat /proc/pressure/memory — some means at least one task was stalled on memory; full means every non-idle task was. Sustained nonzero full is the cleanest “you are thrashing” signal Linux exposes.
• vmstat 1 — the si/so columns show swap traffic in KB/s. Nonzero and sustained means anonymous pages are moving.
• grep -E 'pgfault|pgmajfault' /proc/vmstat — sample twice and diff. The ratio of major to total faults is what you care about.
• ps -o pid,comm,min_flt,maj_flt,rss -p <pid> for per-process fault counts, or perf stat -e page-faults,major-faults ./yourprog for a single run.
• cat /proc/<pid>/smaps_rollup — use Pss and Private_Dirty, not RSS. RSS counts every shared page fully against every process that maps it, so summing RSS across a process tree routinely exceeds physical RAM.
• Inside a cgroup: memory.current, memory.events (the high, max, and oom_kill counters tell you whether you were throttled or killed), and the workingset_refault* counters in memory.stat.
• After the fact: dmesg -T | grep -i 'out of memory' prints the kernel’s task table and the victim it picked.

The fix follows from the reading. High minor faults with flat RSS is normal and cheap — ignore it. High major faults with swap traffic means your working set exceeds RAM; either shrink it or buy more. Repeated 137s with low host pressure means your memory.max is wrong, not your code. And a process whose RSS climbs monotonically across restarts is a leak, which no amount of kernel tuning will fix.

Huge pages are worth naming here because they get recommended for the wrong reason. They reduce TLB misses, which helps pointer-chasing workloads over large heaps. They do not give you more memory, and transparent huge pages can make fragmentation and allocation latency worse under pressure.