Count Unique Elements in Text File with AWK

Counting the distinct/unique elements of text file is a common task.  Below is an example of doing this is AWK, using sample_data_1.txt.

Here is what is happening above:

  • cat sample_data_1.txt – reading the file piping the data to AWK
  • BEGIN{FS=”\t”} – specifying the field separators of the file
  • NR>1 – Only executing the following code block if the record number is greater than 1 (removing the header)
  • names[$2]=1 – This script counts the distinct elements of column number 2. So here we are storing the values of this column in an array.  AWK arrays are associated arrays (holding keys and values).  Each value is simply set to “1” as a place holder for the value.
  • END{print length(names)} – printing the length of the names array

Using an array in AWK is much faster than a common alternative of using the sort and uniq:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">