Illustrated UNIX System V/BSD: Module 149 uniq

Module 149
uniq

DESCRIPTION

The external uniq command reports duplicate lines in a sorted file. It reads in lines and compares the previous line to the current line. Depending on the options specified on the command line it may display only unique lines or one occurrence of repeated lines or both types of lines. The default output is to display lines that only appear once and one copy of lines that appear more than once.

COMMAND FORMAT

Following is the general format of the uniq command.

     uniq [ -cdu [ +n ][ -n ][ input [ output ] ]

Options

The following options are used to control how uniq functions.

-c	Precedes each line with a count of the number of times it occurred in the input.
-d	Deletes duplicate copies. Only one line out of a set of repeated lines is displayed.
-u	Displays only lines not duplicated (uniq lines).
-n	Ignores the first n fields of an input line when comparing for duplicate lines. A field is a string of nonblank characters. A blank is a space or tab.
+n	Ignores the first n characters of an input line when comparing for duplicate lines.

Arguments

The following arguments may be passed to the uniq command.

input	The name of the file containing the input data.
output	The name of the file to hold the output data. If no output file is specified, the output is displayed on the standard output.
	The input and output files must not be the same name. If they are, the contents of the file are destroyed.
	If no input file is specified, uniq reads from the standard input and writes to the standard output.
	You cannot specify an output file without specifying an input file.

RELATED COMMANDS

Refer to the sort command described in Module 122.

RELATED FILES

The uniq command can read from the standard input or from an input file. It writes to the standard output or to an output file.

APPLICATIONS

You can use uniq to reduce duplicate lines from a file. First the file must be sorted, then you can remove the duplicate lines, reducing the size of the file.

It is also useful to filter out multiple blank lines from unsorted or sorted output of other commands. For example, the dircmp command displays its output using pr; thus the output usually scrolls off your screen before you can read it. But if you pipe the output of the dircmp command through the uniq command, the blank lines are reduced and the output is compact.

TYPICAL OPERATION

In this activity you use the uniq command to display a unique set of lines from a file that has already been sorted. Begin at the shell prompt.

1. Type vi unsorted and press Return. Enter the following lines of text, pressing Return after each line.

   cj> vi unsorted
   ia
   b
   c
   c
   b



   a
   z
   y
   x<Esc>ZZ

2. Type uniq unsorted and press Return. Notice the only change to the data is that three blank lines are reduced to one line.

3. Type sort unsorted | uniq and press Return. The output should be sorted alphabetically and a reduced listing compared to the contents of the file.

4. Type rm unsorted and press Return to remove the unsorted file.

5. Turn to Module 123 to continue the learning sequence.

Table of Contents