Linux awk Tutorial: Columns, Filters and Totals

awk is a small language for data in columns. It reads a file line by line, splits each line into fields, and runs your program on them.

Print columns#

By default, awk splits on spaces. $1 is the first field, $2 the second, $NF the last, and $0 the whole line:

echo "Asha 30 Hyderabad" | awk '{print $1, $3}'
ls -l /etc/hostname | awk '{print $1, $NF}'

Asha Hyderabad
-rw-r--r-- /etc/hostname

Set the separator with -F:

awk -F, '{print $1, $4}' staff.csv | head -n 3

name salary
Asha 90000
Ravi 55000

Filter rows#

A condition before the braces selects lines. NR is the line number, so NR > 1 skips the header:

awk -F, 'NR > 1 && $4 > 80000 {print $1, $4}' staff.csv

Asha 90000
Meena 105000
Divya 88000

With no action, awk prints the matching line:

awk -F, '$2 == "Pune"' staff.csv
awk -F, '/Support/' staff.csv

Ravi,Pune,Sales,55000
Divya,Pune,Engineering,88000
Karan,Chennai,Support,42000

Totals and averages#

Variables start at zero. END runs after the last line:

awk -F, 'NR > 1 {sum += $4; n++} END {print "Total:", sum; print "Average:", sum / n}' staff.csv

Total: 441000
Average: 73500

Group by a column#

Arrays can use text as keys, which makes grouping easy:

awk -F, 'NR > 1 {total[$3] += $4} END {for (d in total) print d, total[d]}' staff.csv | sort

Engineering 283000
Sales 116000
Support 42000

This is like SELECT department, SUM(salary) ... GROUP BY department in SQL.

Count things#

awk -F, 'NR > 1 {count[$2]++} END {for (c in count) print count[c], c}' staff.csv | sort -nr

3 Hyderabad
2 Pune
1 Chennai

Formatted output#

printf controls the layout. %-12s is left-aligned text, 12 characters wide, and %8d is a number 8 characters wide:

awk -F, 'NR > 1 {printf "%-8s %8d\n", $1, $4}' staff.csv

Asha        90000
Ravi        55000
Meena      105000
Karan       42000
Divya       88000
Arjun       61000

Built-in variables#

BEGIN runs before the first line, and OFS joins output fields:

awk 'BEGIN {FS=","; OFS=" | "} NR <= 2 {print $1, $2}' staff.csv

name | city
Asha | Hyderabad

Passing shell variables#

Use -v, and do not paste variables into the program:

limit=100000
awk -F, -v limit="$limit" 'NR > 1 && $4 >= limit {print $1}' staff.csv

Meena

When to use which tool#

Common mistakes#

Practice#

awk -F, 'NR > 1 {if ($4 > max[$2]) max[$2] = $4} END {for (c in max) print c, max[c]}' staff.csv | sort

Chennai 42000
Hyderabad 105000
Pune 88000

Key takeaway#

Next: File Permissions.