Where grep finds lines and sed edits them, awk understands that a line has columns. That one idea is why it survives: any time your data has fields — log files, CSVs, the output of ps or df — awk lets you pick them out, do arithmetic on them, and summarise them, in a single line.
It is a full programming language, which scares people off. You can ignore almost all of it. Ninety per cent of real use is printing a field or adding up a column.
Fields
awk splits every line on whitespace and numbers the pieces from 1:
echo "alice 34 engineer" | awk '{print $2}'
34| Variable | Meaning |
|---|---|
$0 | The whole line |
$1, $2, … | Individual fields |
NF | Number of fields on this line |
$NF | The last field — extremely useful |
NR | Record number, i.e. the current line number |
FS | Input field separator (default: whitespace) |
OFS | Output field separator (default: a space) |
$NF is worth committing to memory. It gets you the last column without knowing how many there are, which matters because log lines are rarely uniform.
Changing the separator
awk -F: '{print $1}' /etc/passwd # colon-separated
awk -F, '{print $3}' data.csv # comma
awk -F'\t' '{print $2}' data.tsv # tabDefault whitespace splitting is smarter than a literal space: it collapses runs of spaces and tabs, and ignores leading whitespace. That is why ps aux | awk '{print $2}' works on neatly aligned columns without any fiddling.
Patterns and actions
An awk program is a list of pattern { action } pairs. Omit the pattern and the action runs on every line; omit the action and matching lines are printed.
awk '/error/' # print lines matching "error" (like grep)
awk '/error/ {print $1}' # first field of matching lines only
awk '$3 > 100' # lines where field 3 exceeds 100
awk '$1 == "root" {print $NF}' # last field where field 1 is exactly root
awk 'NR > 1' # skip a header row
awk 'NF == 0' # blank lines
awk 'length($0) > 80' # lines longer than 80 charactersThat third example is the thing awk does that grep cannot: compare a field numerically. A regex has no idea that 100 is bigger than 99.
BEGIN and END
Two special patterns: BEGIN runs before the first line, END after the last. END is where summaries happen.
# Sum a column
awk '{sum += $3} END {print sum}' data.txt
# Average
awk '{sum += $1} END {print sum/NR}' numbers.txt
# Count lines matching a condition
awk '/timeout/ {n++} END {print n+0}' app.log
# Add a header, then the data
awk 'BEGIN {print "USER\tSHELL"} {print $1 "\t" $7}' FS=: /etc/passwdNote print n+0 in the third example. If nothing matched, n was never set, and printing it alone gives an empty line rather than 0. Adding zero forces it to a number — a small trick that saves confusion in scripts.
Associative arrays
This is where awk stops being a field printer and starts being genuinely powerful. Arrays are indexed by strings, and you get grouping and counting almost for free.
# Count requests per IP in an access log
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn
# Total bytes transferred per status code
awk '{bytes[$9] += $10} END {for (code in bytes) print code, bytes[code]}' access.log
# Count shells in use
awk -F: '{shells[$7]++} END {for (s in shells) print shells[s], s}' /etc/passwd | sort -rnThe equivalent with sort | uniq -c works for simple counting, but the moment you need to sum a second column per group, awk is the shortest correct answer.
Formatting output
awk '{print $1, $3}' # comma inserts OFS (a space)
awk '{print $1 "-" $3}' # no comma: joined with no separator
awk -v OFS=, '{print $1, $2}' # output as CSV
awk '{printf "%-20s %8.2f\n", $1, $2}' # aligned columns, 2 decimal placesprintf follows the C convention and does not add a newline, so you must include \n yourself. -v sets a variable before the program runs, which is also how you pass shell values in safely:
threshold=500
awk -v t="$threshold" '$3 > t' data.txtOne-liners worth keeping
# Disk usage above 80% (see the disk space page)
df -h | awk 'NR>1 && $5+0 > 80 {print $6, $5}'
# The PIDs of every nginx process
ps aux | awk '/[n]ginx/ {print $2}'
# Top 10 IPs hitting a site
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# Print columns 2 to the end
awk '{$1=""; sub(/^ /, ""); print}' file.txt
# Remove duplicate lines without sorting first
awk '!seen[$0]++' file.txt
# Sum the sizes reported by ls
ls -l | awk '{total += $5} END {printf "%.1f MB\n", total/1024/1024}'awk '!seen[$0]++' deserves a moment. It is the shortest way to deduplicate a file while preserving order, which sort -u cannot do. The array records each line it has seen; the ! makes the first occurrence true (printing it) and every later one false.
Gotchas
Use single quotes
awk’s $1 and the shell’s $1 look identical. Inside double quotes the shell substitutes first and awk receives an empty string. Always wrap the program in single quotes, and pass shell values with -v.
Numbers and strings are the same thing, until they are not
awk decides on context, which occasionally surprises. $5 > 80 on a column containing 90% compares strings, so "9" > "8" happens to give the right answer while "100%" > "80%" does not. Force numeric comparison with $5+0 > 80.
Modifying a field rewrites the line
Assigning to $1 causes awk to rebuild $0 using OFS, which collapses your original spacing. If you were relying on the input alignment, it is gone.
Do not parse CSV with quoted commas
-F, splits on every comma, including ones inside "quoted, fields". It works on your sample and breaks on real data. Use a real CSV tool — csvkit, miller (mlr), or Python — when quoting is involved.
Know when to stop
awk has functions, loops and recursion, and people have written remarkable things in it. But if your one-liner has grown past a couple of lines and you find yourself reaching for string functions, a Python script will be easier to read in six months. Using awk well includes knowing where it ends.
Quick reference
awk '{print $2}' f # second field
awk '{print $NF}' f # last field
awk -F: '{print $1}' f # custom separator
awk 'NR > 1' f # skip header
awk '$3 > 100' f # numeric filter
awk '/pattern/ {print $1}' f # regex plus field
awk '{s += $2} END {print s}' f # sum a column
awk '!seen[$0]++' f # dedupe, keep order
awk -v OFS=, '{print $1,$2}' f # CSV output
awk '{printf "%-10s %5d\n", $1, $2}' f # aligned outputRelated commands
- grep — finding lines; awk takes over when you need the columns.
- sed — editing lines; awk when the edit depends on a field’s value.
cut— simpler and faster for plain column extraction with a fixed delimiter.sortanduniq— the usual companions in a counting pipeline.mlr(Miller) — awk-like processing that genuinely understands CSV and JSON.