Guide1 min readUpdated June 29, 2026

How to remove duplicate lines from text

Short answer

To remove duplicate lines, keep the first time each line appears and drop any later exact copies. Decide up front whether matching should ignore case and surrounding whitespace, because those choices change what counts as a duplicate.

What counts as a duplicate

Deduplication keeps the first occurrence of each line and removes exact repeats. The catch is defining 'exact'. By default Apple and apple (with a trailing space) are different lines, so the option to trim whitespace and ignore case often matters more than the dedupe itself.

Case sensitivity: should Apple and apple be treated as the same?
Whitespace: should leading and trailing spaces be ignored?
Order: keep the original order, or sort the result?
Blank lines: drop them, or keep one?

apple        apple
banana   ->  banana
apple        cherry
cherry

Before and after

Try it: Remove Duplicate LinesRemove duplicate lines locally, with options for case and whitespace.Open tool

Dedupe then cleanFor messy data like CSV exports, trimming and normalizing first makes deduplication far more accurate. See related tools below.

References

uniq command on Wikipedia — Wikipedia

Questions

Does removing duplicates change the order of my lines?

By default it keeps the original order and removes only later repeats. If you sort the output instead, the order changes but the set of unique lines stays the same.

Why are two lines that look identical not treated as duplicates?

They probably differ in invisible ways, such as trailing spaces or different capitalization. Enable trim and ignore-case options to match them.

Does this tool upload my text?

No. Lines are compared locally in your browser and your text is never uploaded.