grep, sed, awk and jq: the four text tools you will use every week

Log files, config files, CSV exports and API responses are all text. Four tools cover nearly everything you need to do with them: grep selects lines, sed transforms lines, awk works with fields and sums, and jq queries and reshapes JSON. Learn ten forms of each and you stop writing throwaway Python for every small job.

grep: find lines

grep 'ERROR' app.log                 # lines containing ERROR
grep -i 'timeout' app.log            # case-insensitive
grep -n 'ERROR' app.log              # with line numbers
grep -c 'ERROR' app.log              # count matching lines
grep -v 'DEBUG' app.log              # lines NOT matching
grep -w 'user' app.log               # whole word only
grep -E 'ERROR|WARN' app.log         # extended regex, alternation
grep -F '1.2.3.4' app.log            # fixed string, dots are literal
grep -o '[0-9]\{1,3\}\(\.[0-9]\{1,3\}\)\{3\}' app.log   # print only the match
grep -r --include='*.py' 'import requests' .                  # recursive
grep -l 'TODO' *.sh                  # only file names
grep -B2 -A5 'Traceback' app.log     # 2 lines before, 5 after

Quote patterns in single quotes so the shell does not touch them. -E gives you +, ?, | and () without backslashes. For large source trees, ripgrep (rg) is a faster drop-in that skips ignored files by default.

sed: edit a stream of lines

sed 's/http:/https:/' urls.txt           # first match per line
sed 's/http:/https:/g' urls.txt          # every match
sed -n '10,20p' app.log                  # print only lines 10 to 20
sed -n '/^ERROR/p' app.log               # print only matching lines
sed '/^#/d; /^$/d' config.conf           # delete comments and blank lines
sed -E 's/([0-9]+)ms/\1 ms/g' app.log    # extended regex with a group
sed 's|/old/path|/new/path|g' file       # any delimiter works
sed -i.bak 's/DEBUG=1/DEBUG=0/' app.env  # edit in place, keep a .bak copy

Careful: sed -i rewrites the file. GNU sed accepts -i alone; the BSD sed on macOS needs an argument, so write sed -i '' 's/a/b/' file there, or use -i.bak which works on both. Try the command without -i first and look at the output.

awk: columns, sums and small reports

awk reads each line, splits it into fields $1, $2… on whitespace (or on the separator given with -F) and runs your program for every line that matches an optional pattern. $0 is the whole line, NF the number of fields and NR the line number.

awk '{print $1}' access.log                       # first column
awk -F: '{print $1, $7}' /etc/passwd              # user and shell
awk '$9 == 500' access.log                        # lines where column 9 is 500
awk '$3 > 1000 {print $1, $3}' sizes.txt          # filter and project
awk '{sum += $5} END {print sum}' access.log      # sum a column
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head
awk 'NR % 2 == 0' file                            # every second line
awk -F, 'NR > 1 {print $2}' data.csv              # skip the header
awk '{printf "%-20s %8.2f\n", $1, $2}' prices.txt # formatted output
awk 'length($0) > 120' *.py                       # long lines

For real CSV with quoted commas, awk’s -F, is not enough; use a CSV-aware tool such as Python’s csv module, mlr (Miller) or DuckDB.

jq: the same idea for JSON

jq (jqlang/jq) reads JSON and applies a filter. . is the whole document, .key selects a field, .[] iterates an array and | chains filters, just like a shell pipe.

curl -s https://api.example.com/users | jq .                 # pretty-print
jq '.name' user.json                                         # one field
jq -r '.name' user.json                                      # raw string, no quotes
jq '.users[].email' data.json                                # one field from each item
jq '.users[] | select(.active == true) | .name' data.json    # filter items
jq '.users | length' data.json                               # count
jq '.users | map(.age) | add / length' data.json             # average
jq -r '.users[] | [.name, .email] | @csv' data.json          # to CSV
jq -r '.users[] | "\(.name)\t\(.email)"' data.json          # to TSV with string interpolation
jq '{total: .count, first: .items[0].id}' data.json          # build a new object
jq -c '.items[]' data.json                                   # one compact object per line
jq -S . config.json                                          # sort keys, good for diffs
jq --arg u alice '.users[] | select(.name == $u)' data.json  # pass a shell variable in
jq -s 'add' part1.json part2.json                            # merge files (slurp)
jq 'keys' config.json                                        # list top-level keys
jq '.. | .id? // empty' data.json                            # every id at any depth

• -r prints strings without quotes, which is what you want when feeding results to other commands.
• -e makes the exit code non-zero when the result is null or false, useful in scripts.
• --arg name value passes a string and --argjson name value passes a number or object; never splice shell variables into the filter text.
• -S sorts keys; -c prints compact; --tab and --indent n control pretty-printing.

Combining them

# Top 10 client IPs behind 5xx responses
awk '$9 ~ /^5/ {print $1}' access.log | sort | uniq -c | sort -rn | head

# Which pods restart the most
kubectl get pods -A -o json | jq -r '.items[] | "\(.status.containerStatuses[0].restartCount)\t\(.metadata.namespace)/\(.metadata.name)"' | sort -rn | head

# Replace a value in every YAML file that mentions it, keeping backups
grep -l 'replicas: 1' *.yaml | xargs sed -i.bak 's/replicas: 1/replicas: 2/'

Official documentation

• GNU grep manual
• GNU sed manual and GNU awk manual
• jq manual