Reshape text

The last chapter taught you how to join things up. This chapter grows what there is to join to.

Nearly every tool on the command line has the same shape: it takes text and gives back text. Which is why joining a tool that sorts to a tool that drops duplicates gets you a summary.

• sort — put things in order (as numbers, on a column you pick, in reverse)
• uniq — drop duplicates that are next to each other (which is why sort comes first)
• cut — pull out a column
• tr — swap characters, delete them
• paste and nl — line things up sideways, number the lines
• sed — swap text, pull lines out
• awk — calculate on columns
• diff — show what differs between two files

sed and awk are small programming languages in their own right, each deep enough for a book. This chapter covers only the shapes you use most — swapping for sed, pulling out columns and adding up for awk. That alone covers nine days out of ten.

Note that uniq can only remove lines that are sitting next to each other. Not knowing that leaves you thinking it is broken, so this chapter has you walk into it on purpose.