Pipes and redirection: connecting commands

Every command you run in a terminal has three standard streams: stdin (input, number 0), stdout (normal output, number 1) and stderr (error messages, number 2). By default stdin is your keyboard and both outputs go to your screen. Pipes and redirection let you rewire those streams.

Pipes: |

The pipe character | takes the stdout of the command on its left and feeds it as stdin to the command on its right. This is the heart of the Unix philosophy: small tools that each do one job, chained together.

# How many files are in this folder?
ls | wc -l

# Show only running processes that mention python
ps aux | grep python

# The 5 most common lines in a log file
sort app.log | uniq -c | sort -rn | head -n 5

Read a pipeline from left to right: each stage receives the previous stage’s output. Only stdout goes through the pipe. Error messages still appear on your screen, which is usually what you want.

Redirecting output to files

• > file writes stdout to a file, replacing whatever was there.
• >> file appends stdout to the end of a file.
• 2> file writes stderr (errors) to a file.
• > file 2>&1 sends both stdout and stderr to the same file.
• < file reads stdin from a file instead of the keyboard.

echo "first line" > notes.txt     # create or overwrite
echo "second line" >> notes.txt   # append

ls missing-folder 2> errors.txt    # save only the error
./build.sh > build.log 2>&1         # save everything

wc -l < notes.txt                   # read input from a file

Order matters in > file 2>&1. It means “point stdout at the file, then point stderr wherever stdout points now”. If you write 2>&1 > file, stderr is copied to the screen before stdout moves, so errors still show up in the terminal. In bash you can also write &> file as a shortcut for both.

Throwing output away

/dev/null is a special file that discards anything written to it. Use it to silence noisy commands.

find / -name "*.conf" 2>/dev/null   # hide "Permission denied" noise
some-command > /dev/null 2>&1        # run silently

Seeing output and saving it: tee

tee copies its input to a file and to stdout at the same time, so you can watch a long command while also keeping a log. Add -a to append instead of overwrite.

npm test 2>&1 | tee test-output.txt

Exit codes and chaining

Every command finishes with an exit code: 0 means success, anything else means failure. You can see the last one with echo $?. Use it to chain commands conditionally:

• a && b runs b only if a succeeded.
• a || b runs b only if a failed.
• a ; b runs b no matter what.

mkdir -p build && cd build
grep -q "TODO" main.py || echo "no TODOs left"

By default a pipeline’s exit code is that of its last command, so false | true counts as success. In scripts, set -o pipefail changes this so any failing stage makes the whole pipeline fail.

A common trap

Never redirect a command’s output into the same file it is reading: sort data.txt > data.txt empties the file before sort even starts, because the shell sets up the redirection first. Write to a new file and rename it, or use a tool’s own in-place option.