Throughput vs Latency: Two Different Questions

People say “the system is slow” and mean two completely different things. Sometimes a single request takes too long. Sometimes each request is fine, but the system can’t keep up with the volume. Those are separate problems with separate fixes, and conflating them is one of the most common sources of wasted performance work.

Two measurements, two questions

Latency is the time a single operation takes from start to finish. You measure it in units of time per operation: 40 ms per request, 8 ms per disk read, 200 ms per page load. Latency answers “how long do I wait for this one thing?”

Throughput is how many operations complete in a given window of time. You measure it the other way around: requests per second, transactions per minute, gigabytes per hour. Throughput answers “how much total work can the system get through?”

The classic way to feel the difference is a highway. Latency is how long it takes one car to drive from the on-ramp to the exit. Throughput is how many cars pass the exit per hour. Widening the road from two lanes to six dramatically increases throughput — far more cars per hour — but it does nothing for the travel time of any single car. A car still drives the same distance at the same speed. Conversely, raising the speed limit cuts each car’s travel time (lower latency) without necessarily moving many more cars per hour.

So the two are related but not the same number, and improving one does not automatically improve the other.

How the trade-offs actually work

Three common techniques each push on this trade-off in a different direction.

Batching groups many operations together and processes them as one unit. A database that flushes 1,000 writes in a single transaction does far more work per second (high throughput) than committing each write individually. But any single write now has to wait for the batch to fill or for a timer to fire before it lands — so its latency goes up. Batching trades latency for throughput.

Parallelism runs operations at the same time on separate workers. Adding more lanes, more CPU cores, or more server replicas raises throughput because more work happens concurrently. It usually doesn’t lower the latency of a single operation — one request still takes as long as it always did — but it stops requests from queueing behind each other, which keeps latency from getting worse under load.

Pipelining breaks an operation into stages and keeps every stage busy. Think of an assembly line, or a CPU instruction pipeline: while stage 3 works on item A, stage 2 works on item B and stage 1 on item C. The total time for one item (latency) is unchanged or even slightly higher, but completed items come off the end much more frequently, so throughput climbs.

A useful rule of thumb connecting them is Little’s Law: in a stable system, the average number of items in flight equals throughput multiplied by average latency. One practical consequence is that if you want more throughput out of a fixed-latency operation, you need more requests in flight at once — that is, more concurrency.

Which one should you optimize?

It depends entirely on the workload. For anything interactive — an API call, a search box, a page load — users feel latency, and specifically tail latency (the slow 1% of requests, often reported as p99). A median of 50 ms means little if one request in a hundred takes three seconds. For bulk and background work — log processing, video encoding, analytics pipelines — nobody is staring at an individual item, so throughput is what matters, and you’ll happily accept higher per-item latency to get more items done per hour.

The mistake to avoid is optimizing the metric that’s easy to measure instead of the one users feel. Adding more replicas makes a dashboard’s throughput number look great while a single slow query still makes the page feel sluggish. Always tie the metric back to the actual experience.

FAQ