Unix Pipes: The One-Line Superpower Behind the Command Line

The pipe character — a single | — is one of the smallest pieces of syntax in any shell, and one of the most powerful. It lets you snap independent programs together into a custom data-processing line that you assemble on the spot, no script required.

Three streams every program is born with

When a process starts on a Unix-like system, the kernel hands it three open file descriptors by default:

• stdin (descriptor 0) — where it reads input.
• stdout (descriptor 1) — where it writes normal output.
• stderr (descriptor 2) — where it writes errors and diagnostics.

By default all three are wired to your terminal: you type into stdin, and both stdout and stderr appear on screen. The key insight is that these are just connection points. A program writing to stdout does not know or care whether the other end is a terminal, a file, or another program. That indifference is exactly what makes pipes possible.

A pipe rewires stdout. When you write A | B, the shell connects the stdout of A to the stdin of B. Command A thinks it is printing to the screen; command B thinks someone is typing at it. Neither needs to be aware of the other. The shell sets up the plumbing between them.

ps aux | grep node

Here ps aux lists every running process to its stdout. Instead of reaching your screen, that text flows into grep node, which passes through only the lines containing node. Two programs that know nothing about each other, cooperating through a stream of bytes.

The Unix philosophy: small tools that compose

Pipes are the practical expression of an old design idea: write programs that do one thing well, and make them work together by passing text streams. Rather than one giant program with a “search, then count, then rank” feature, you get separate tools — grep, sort, uniq, wc — each tiny, each replaceable, each combinable in ways their authors never anticipated.

A classic example ranks the most frequent values in a log. Say you want the top IP addresses in an access log:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Read it left to right as an assembly line: awk pulls the first field (the IP) from each line; sort groups identical IPs next to each other; uniq -c collapses runs of duplicates and prefixes each with its count; sort -rn sorts those counts in reverse numeric order; head keeps the top few. (uniq only collapses adjacent duplicates, which is why the first sort is required.) Five general-purpose tools, zero custom code, and you have a working frequency analysis.

That is the payoff. Each tool stays simple, and the combinations are effectively unlimited.

Two things people get wrong

Pipes stream; they don’t batch. A pipeline does not run A to completion, save everything, then start B. All commands launch at roughly the same time and run concurrently. As A produces output, B consumes it. This is why tail -f log | grep ERROR works on a file that never ends — tail keeps emitting lines and grep keeps filtering them live. A pipe has a small in-kernel buffer; if B reads slowly, the buffer fills and A is paused until there is room (backpressure). Memory stays bounded even on enormous inputs.

stderr is not piped. A pipe only redirects stdout. Error messages go to stderr, which stays attached to your terminal and sails right past the next command. So this misses errors:

make 2>&1 | grep -i warning

Without 2>&1, warnings printed to stderr would never reach grep. The 2>&1 means “redirect descriptor 2 (stderr) to wherever descriptor 1 (stdout) currently points” — folding errors into the stream so the pipe carries them too. Order matters: it has to come before the pipe sees the output.