What a JIT Compiler Actually Does at Runtime, and Why It Beats an Interpreter

Your Python, JavaScript, and Java code does not run on your CPU. It runs on a program that reads it and does what it says. A just-in-time compiler sits next to that program, watches which parts execute often, and replaces the hot parts with real machine code while the process is still live.

That’s the one-sentence version. The details are the useful part, because they explain both why JIT-compiled code pulls away from interpreted code and why your service is measurably slower for the first thirty seconds after every deploy.

The tax an interpreter pays on every instruction

Take a + b inside a loop. Here is what a bytecode interpreter does for that one operation, every single iteration:

1. Fetch the next bytecode from the instruction stream.
2. Dispatch — an indirect jump into the handler for that opcode.
3. Pop two operands off the value stack, or read two slots from the frame.
4. Check their runtime types. Both small integers? Both doubles? Is one a string, making this concatenation? Has __add__ been overridden?
5. Do the addition. One CPU instruction.
6. Box the result into a heap object and push it back.

Step 5 is the work. Steps 1, 2, 3, 4, and 6 are overhead, and they repeat forever.

Two of those hurt more than the rest. The dispatch in step 2 is an indirect branch whose target changes constantly, which is close to the worst case for a CPU branch predictor — this is why serious interpreters use computed-goto threaded dispatch instead of a switch, giving the predictor one branch site per opcode instead of one for the whole loop. And step 6 allocates. A loop that adds two numbers a million times can allocate a million objects, each of which the GC then has to trace and free.

The galling part is that the answers to step 4 were identical all million times. The interpreter re-derives them anyway, because it has no memory.

CPython 3.11 attacked exactly this with its specializing adaptive interpreter (PEP 659). After a generic BINARY_OP executes a few times with two integers, the interpreter rewrites that bytecode in place to a specialized BINARY_OP_ADD_INT handler that skips most of the type dispatch, guarded by a cheap check that falls back if the assumption breaks. That is the core JIT idea — observe, specialize, guard — implemented without emitting a byte of machine code.

What a JIT does while your code is running

A real JIT does four things, roughly in this order.

Profiling. Counters, mostly. Per-function invocation counters and per-loop back-edge counters. When a counter crosses a threshold, that code is “hot” and gets queued for compilation. HotSpot’s non-tiered CompileThreshold historically defaulted to 10,000 invocations for the server compiler; the tiered compilation that ships by default now uses several lower thresholds instead. The exact numbers matter less than the shape: nothing gets compiled until it has proven it’s worth compiling.

Tiering. There isn’t one compiler, there’s a ladder. V8 runs Ignition (a bytecode interpreter), then Sparkplug (a baseline compiler that emits machine code fast and does no type analysis), then Maglev (a mid-tier optimizer), then TurboFan (the full optimizer, slow to run, best output). HotSpot runs interpreter, then C1, then C2. Each rung trades compile time against code quality. Code that runs 200 times gets the cheap tier; code that runs 200 million times earns the expensive one.

Speculation. This is the move that actually wins. The profile says: at this call site the receiver has had the same hidden class every time; this variable has been a 32-bit integer every time. The optimizer does not prove those facts — it assumes them, emits a guard (one compare-and-branch), and compiles everything after the guard as if the code were statically typed.

Now a + b is one add instruction on two machine registers. No fetch, no dispatch, no stack traffic, no type check, no boxing. The six-step sequence from the previous section collapses into step 5.

Inlining, and everything it unlocks. Once a call site is monomorphic and the callee is small, the JIT inlines the body. Inlining isn’t valuable by itself; it’s valuable because it makes every other optimization possible. With the body inlined, escape analysis can prove a freshly allocated object never leaves the frame and delete the allocation entirely. Constants propagate across what used to be a call boundary. Loop-invariant expressions hoist out. Array bounds checks disappear when the compiler can prove i stays below arr.length. An interpreter can do none of this, because to an interpreter every call is an opaque box.

One more mechanism you’ll hit in profiles: on-stack replacement. If a single function call enters a loop that runs ten million times, waiting for the next call to use the optimized code is useless — there may not be one. OSR compiles the loop, reconstructs the running frame in the new code’s layout, and jumps into it mid-flight.

Where the JIT loses

Warmup. Cold code runs interpreted. A CLI tool, a serverless invocation, or a CI job may exit before the optimizing tier ever produces anything, so you pay the profiling and compilation overhead and collect none of the benefit. This is the entire argument for ahead-of-time approaches: GraalVM native-image, class data sharing on the JVM, and checkpoint-restore schemes all exist to skip the ramp.

Resource cost. Compiler threads compete with application threads for cores. Compiled code lives in a fixed-size code cache — HotSpot’s ReservedCodeCacheSize defaults to 240MB under tiered compilation — and profiling metadata occupies heap that your program doesn’t get to use.

Deoptimization. Every speculative assumption is a guard, and guards can fail. Pass a string to a function that has only ever seen integers, and the runtime bails out of the optimized frame, rebuilds interpreter state mid-execution, throws the compiled code away, and starts profiling again.

Benchmarks that measure the wrong thing. A microbenchmark that times the first 100 iterations measures the interpreter. One that times after ten seconds of load measures TurboFan or C2. These can differ by more than an order of magnitude, and the direction of your “optimization” can flip depending on which one you accidentally measured. On the JVM, use JMH with explicit warmup iterations. Elsewhere, discard the first N runs deliberately and say so.