Fulfledge

Reshape text

INPUT · Slides

Drop duplicates, count them (`uniq`)

01 / 06

uniq only looks next door

Let us start with the thing people trip over most.

~ $ uniq fruit.txtappleorangeapplebananaorangeapple

Nothing went, even though there are three apple.

Because uniq only looks at lines next to each other. It reads down from the top and throws a line away if it is the same as the one just before. That is all it does.

There are three apple, but with orange in between they never end up next to each other.

02 / 06

Which is why you sort first

The fix is fixed. Sort first, so the same things end up side by side.

sort fruit.txt | uniq

sort gives you apple, apple, apple, banana, orange, orange, and from there uniq can squash the neighbours.

Learn sort | uniq as a set. There is hardly ever a reason to use uniq on its own.

~ $ sort fruit.txt | uniqapplebananaorange

03 / 06

-c counts them

This is the best thing about uniq. Add -c (count) and you get how many of each there were.

~ $ sort fruit.txt | uniq -c      1 banana      2 orange      3 apple

sort -u only removes them; it gives you no numbers. Only uniq can count.

That, joined to sort -rn, is the classic three stages for summarising. Most of the work of reading logs is settled in that shape.

04 / 06

-d and -u sort them apart

uniq has two flags that face opposite ways.

  • -d (duplicate) — put out only what appeared two or more times
  • -u (unique) — put out only what appeared exactly once
sort fruit.txt | uniq -d   # apple and orangesort fruit.txt | uniq -u   # only banana

Both have proper uses. -d for "find the duplicates" (has the same email address been registered twice?), -u for "find the rare thing that only happened once" (an error that hardly ever appears).

05 / 06

Choosing between this and sort -u

The last lesson said sort -u also drops duplicates. So which do you use?

What you wantHow to write it
just drop duplicatessort -u
the counts as well`sort \uniq -c`
only the duplicates`sort \uniq -d`
only the one-offs`sort \uniq -u`

For dropping alone, sort -u is shorter. Everything else is uniq.

Add -i and it ignores case, for when you want Apple and apple counted as the same thing.

06 / 06

Now have a go

Three files.

  • fruit.txt — six lines of fruit (apple x3, orange x2, banana x1)
  • access.log — six lines of addresses (192.168.0.1 x3, 10.0.0.5 x2, 192.168.0.2 x1)
  • caps.txt — Apple, apple, orange

Think of access.log as a real log, shrunk. Counting "which address do most of them come from" is what people who watch web servers do every day.