Join them with a pipe (`|`)

01 / 07

Joining without going through a file

The way you have done it so far, combining two tools needed a file.

sort fruit.txt > temp.txtuniq temp.txtrm temp.txt

Make somewhere to put it, use it, tidy it away. A nuisance, and thinking up the name is wasted effort.

With a pipe (|) you join them directly, with no file at all.

sort fruit.txt | uniq

02 / 07

`|` wires a way out to a way in

| is the symbol for "wire the way out on the left to the way in on the right".

commandA | commandB

A's standard output (number 1) goes
straight into B's standard input (number 0)

Nothing gets written to disk on the way, so it is fast, and you never have to name anything in between.

And you can join as many as you like. Three stages or five, you write it the same way.

~ $ ls | wc -l4

03 / 07

Only number 1 goes through a pipe

Once more, from the last lesson. Errors (number 2) do not go through a pipe.

ls missing.txt | wc -l      # 0 (the error did not come through)ls missing.txt 2>&1 | wc -l # 1 (merged, so it did)

That is a considerate arrangement. If an error got mixed into the middle of a pipe, the next tool would receive strange text and misbehave. Trouble to the human, data to the next tool — the logic holds.

04 / 07

The classic three stages: count, most first

You may as well learn the most used shape of all: how many of each there are, most first.

sort fruit.txt | uniq -c | sort -rn

One stage at a time, it reads like this.

1. sort — gather the same things next to each other
2. uniq -c — squash the neighbouring duplicates and write the count at the front
3. sort -rn — order by that number, largest first (r for reverse, n for as a number)

Almost all log summarising is this shape. "Which error is commonest", "which address do most of them come from" — this is how you get it.

~ $ sort fruit.txt | uniq -c | sort -rn      3 apple      2 orange      1 banana

05 / 07

Build it up one stage at a time

You do not have to write a long pipe in one go. Write a stage, run it, look at it. Then repeat.

sort fruit.txt                        # see whether it lines upsort fruit.txt | uniq -c              # see whether the counts appearsort fruit.txt | uniq -c | sort -rn   # see whether the order changes

There is a big advantage to working this way: you can tell where it stopped doing what you expected.

Write five stages at once and get something odd, and you have no idea which stage is at fault. One at a time, the culprit is the stage you just added.

06 / 07

Stopping partway is fine

Here is a fun experiment.

yes | head -3

yes is a command that puts out y forever. And yet this stops.

When head -3 has taken three lines and finished, yes gets told that where it was writing has closed, and finishes too.

That mechanism is what makes huge log | head safe to use. You do not have to wait for the whole thing to be read.

07 / 07

Now have a go

fruit.txt holds six lines of fruit names: three apple, two orange and one banana.

This lesson uses the same file over and over. What to watch is how what comes out changes as you add stages.

There are a few new tools too: uniq (squash duplicates), seq (line up numbers) and ps (see what is running). Each gets proper treatment in a later chapter, so here you only need to meet them as something to join to.