AWK Remove Duplicates: Lines, Columns and CSV Rows

The shortest way to remove duplicate lines from a file in Linux is an AWK one-liner that most people copy without ever reading. It is worth understanding, because the same trick handles duplicates by column, duplicate CSV rows, keeping the last occurrence instead of the first, and listing only the lines that repeat. This guide covers each case with sample input and the exact output.

The quick answer

awk '!seen[$0]++' file.txt

This prints each distinct line the first time it appears and drops every later copy. Unlike sort -u, it keeps the original order of the lines and does not need the file to be sorted.

Sample files

printf 'apple\nbanana\napple\ncherry\nbanana\napple\n' > dup.txt

cat > emails.csv <<'EOF'
id,email,city
1,ann@x.com,Colombo
2,bob@x.com,Kandy
3,ann@x.com,Galle
4,cy@x.com,Kandy
5,bob@x.com,Colombo
EOF

# used further down
printf 'a\na\nb\na\nc\nc\n' > con.txt
printf 'Apple\napple\nAPPLE\nbanana\n' > case.txt
printf 'apple\napple \nbanana\n' > sp.txt
printf 'x\r\nx\nx\r\n' > crlf.txt

Remove duplicate lines and keep the order

awk '!seen[$0]++' dup.txt
# apple
# banana
# cherry

sort -u dup.txt gives the same three lines here, but only because they happen to be in alphabetical order already. sort -u always returns sorted output; the AWK version returns the lines in the order they first appeared.

How !seen[$0]++ works

Read it from the inside out:

1. $0 is the whole current line, used as a key in the array seen.
2. seen[$0]++ adds one to that entry, but the value of the expression is the number before the increment. A key that has never been used counts as 0.
3. ! turns that into true when it was 0 (first time) and false for 1, 2, 3... (a repeat).
4. An AWK rule with a condition and no braces prints the line when the condition is true.

So the first apple sees 0, passes, and is printed. The second sees 1 and is skipped. The condition is true only for the first copy of each line.

Remove duplicates by one column

Use the column as the key instead of the whole line. This keeps the first row for each email address:

awk -F, '!seen[$2]++' emails.csv
# id,email,city
# 1,ann@x.com,Colombo
# 2,bob@x.com,Kandy
# 4,cy@x.com,Kandy

Rows 3 and 5 are gone because their emails had already appeared. -F, sets the comma as the field separator (see AWK field separators). The header row was kept only because no earlier line had the key email. If a data row could ever match the header text, protect the header explicitly:

awk -F, 'NR==1 || !seen[$2]++' emails.csv

Remove duplicates by several columns

Join the columns with a comma inside the brackets. AWK builds one key from them:

printf 'a,1\na,1\na,2\nb,1\n' | awk -F, '!seen[$1,$2]++'
# a,1
# a,2
# b,1

Two rows count as duplicates only when both columns match. a,1 and a,2 share the first column but are different rows.

Keep the last occurrence instead of the first

Reverse the file, keep the first of each, and reverse it back. tac prints a file from the last line to the first:

tac emails.csv | awk -F, '!seen[$2]++' | tac
# id,email,city
# 3,ann@x.com,Galle
# 4,cy@x.com,Kandy
# 5,bob@x.com,Colombo

Each email now appears once, with its last row. The result is ordered by where each row last appeared, not by where it first appeared. tac is part of GNU coreutils; on macOS use tail -r instead.

Remove only consecutive duplicates

This is what uniq does. Compare each line with the previous one:

awk '$0 != prev {print} {prev = $0}' con.txt

On a file with the lines a a b a c c, this prints a b a c. The second a after b stays, because it is not next to another a. Plain uniq gave the same result. Unlike !seen[$0]++, this uses almost no memory because it only remembers one line.

Show only the duplicates

awk 'seen[$0]++' dup.txt             # every repeated copy (2nd, 3rd, ...)
# apple
# banana
# apple

awk 'seen[$0]++ == 1' dup.txt        # each duplicated value, once
# apple
# banana

Without the !, the condition is true for the second and later copies. Adding == 1 selects only the second copy, so each duplicated value is printed once.

Show lines that appear only once

This needs two passes, so name the file twice. The first pass counts, the second prints the lines whose count is 1:

awk 'NR==FNR {c[$0]++; next} c[$0] == 1' dup.txt dup.txt
# cherry

NR==FNR is true only while AWK reads the first copy of the file. next skips to the following line without running the second rule.

Count how many times each line appears

awk '{c[$0]++} END {for (k in c) print c[k], k}' dup.txt | sort -k1,1nr -k2
# 3 apple
# 2 banana
# 1 cherry

This is the same technique as in counting occurrences with AWK.

Ignore case or stray whitespace

Two lines that look the same to you are different keys to AWK if their case, trailing spaces or line endings differ. Normalise the key, and keep the original line for printing:

awk '!seen[tolower($0)]++' case.txt          # Apple, apple, APPLE, banana  ->  Apple, banana
awk '{sub(/[ \t]+$/, "")} !seen[$0]++' sp.txt   # "apple" and "apple " become one line

In the first command the key is lowercase, but the line printed is the original one, so the first spelling (Apple) is kept. In the second, sub() strips trailing spaces from the line before the duplicate check, and the cleaned line is what gets printed.

A file from Windows has a hidden carriage return at the end of each line. A test file with x\r\n, x\n and x\r\n produced 2 lines with the plain command and 1 after removing the carriage return:

awk '{sub(/\r$/, "")} !seen[$0]++' crlf.txt

Remove duplicates across several files

awk '!seen[$0]++' file1.txt file2.txt > merged.txt

The seen array is shared across all the files in one run, so a line from the second file that already appeared in the first is dropped.

Save the result

Never redirect output into the file you are reading, as in awk '...' file > file. The shell empties the file before AWK starts, and you lose the data. Write to a new file and move it into place:

awk '!seen[$0]++' dup.txt > dup.clean && mv dup.clean dup.txt

With gawk you can edit in place instead. I ran this on gawk 5.2.1:

gawk -i inplace '!seen[$0]++' dup.txt

-i inplace is a gawk feature. mawk does not have it, so use the temporary file method there.

Count how many lines were removed

awk '!seen[$0]++ {kept++} END {print NR - kept, "removed"}' dup.txt
# 3 removed

AWK, sort -u or uniq?

The AWK array holds every distinct line in memory. For a file with millions of mostly unique lines that can be large, and sort -u is the safer choice. For the usual log, list or CSV, AWK is fast and simple.

Common problems

• "Duplicates" that are not removed. Trailing spaces, different case or Windows line endings make lines differ. Normalise the key as shown above. cat -A file shows hidden characters.
• The header disappeared or was duplicated. Use NR==1 || !seen[$2]++ to always keep it.
• Quoted CSV fields. -F, splits inside quotes, so a value like "Smith, John" becomes two fields and the wrong key is used. For quoted CSV, deduplicate on a column that has no commas, or use a CSV-aware tool.
• The file came out empty. You redirected into the input file. Write to a new file first.

Frequently asked questions

How do I remove duplicate lines in Linux without sorting?

Use awk '!seen[$0]++' file. It keeps the first copy of each line and preserves the original order.

How do I remove duplicates based on one column in AWK?

Use that column as the array key, for example awk -F, '!seen[$2]++' file.csv to keep the first row for each value in column 2.

How do I keep the last duplicate instead of the first?

Reverse the file, deduplicate, and reverse again: tac file | awk '!seen[$0]++' | tac.

What does !seen[$0]++ mean?

It is true only the first time a line is seen. seen[$0]++ returns the previous count, and the ! turns a count of 0 into true, so the default action (print) runs for first occurrences only.

Is the AWK method case sensitive?

Yes. Use !seen[tolower($0)]++ to treat Apple and apple as the same line.

Next in this series

• AWK Command in Linux: A Beginner's Guide with Examples
• AWK Count Occurrences: Lines, Words, Characters and Values
• How to Sum a Column with AWK
• AWK Cheat Sheet