Calculate on columns (`awk`)

The most basic shape of awk is this.

awk '{print $2}' gaps.txt

$1 is column 1, $2 is column 2 and so on. $0 is the whole line.

Inside the {} you write what to do with each line. awk reads the file a line at a time and runs the {} for every line.

And it does not care about the width of the spacing. Whether it is tanaka 80 or suzuki 95, $2 is the score. That is the big difference from cut.

For comma separated data like a CSV, name the separator with -F (the F of field separator).

awk -F, '{print $1}' grades.csv

It is the equivalent of cut -d. Different name, same job.

Being able to reorder columns is another difference from cut.

awk -F, '{print $2, $1}' grades.csv     # 80 tanakaawk -F, '{print $3 "," $1}' grades.csv  # 3,tanaka

List things in print with commas and they are joined by a space; put "," between them and they are joined by a comma. Remember cut -f2,1 refusing to reorder? With awk you are free.

awk gives you some handy values.

awk '{print NR, $1}' gaps.txt     # number the linesawk '{print $NF}' access.log     # just the last column

$NF is worth remembering. Even in data where the number of columns varies line by line, you can still get the last one.

With NR you have a stand-in for nl too. Several tools doing the same job is a familiar sight by now.

Put a condition before the {} and only the matching lines get processed.

awk -F, '$2 > 80 {print $1}' grades.csv    # names of those above 80awk -F, '$1 == "sato" {print $2}' grades.csvawk -F, '/tanaka/ {print $2}' grades.csv   # lines containing tanakaawk -F, 'NR == 2 {print $1}' grades.csv    # line 2

Did you notice this goes beyond grep? grep could only search as text, but awk can compare numbers, as in "column 2 is greater than 80".

Leave out the {print} and it means "put the whole line out". So awk -F, '$2 > 80' grades.csv on its own narrows the data down.

This is what awk is really worth having for: a total.

awk -F, '{sum += $2} END {print sum}' grades.csv

Read it like this.

• {sum += $2} — on each line, add column 2 to sum
• END {print sum} — after every line has been read, put sum out

END is where you write "do this once at the end". BEGIN is the opposite, "once before starting" (used for putting a heading on).

You can get the average too, because NR holds the number of lines.

awk -F, '{sum += $2} END {print sum/NR}' grades.csv

Did you notice there is no declaration setting sum up? In awk a variable starts at 0 the moment you start using it.

Three files.

• grades.csv — three lines of name,score,rank
• gaps.txt — uneven spacing, as in tanaka 80
• access.log — three lines of address method path result (a real log, shrunk)

Do not forget the single quotes. $ is special to the shell as well, and without quotes your $1 disappears.

awk is deep enough for a book of its own. This lesson only covers pulling out columns and calculating, which is still enough for most day to day work.