Remove duplicate lines
Paste a list, get every line once, with the removed duplicates counted.
Related tools
Paste a list and every line that has already appeared is dropped, while the first one stays where it was. The counts under the boxes show how many lines went in, how many came out, and how many duplicates were removed, so you can see what the clean-up cost. Use it on an email list, a keyword export, a log file, a set of URLs, or a spreadsheet column that grew copies during a merge. Four options decide what counts as the same line: case, spaces around the line, blank lines and the sort order. The list stays in your browser and is never uploaded.
How to use
- Paste or type your list in the left box, one item per line.
- Turn off case sensitivity if Apple and apple should count as the same item.
- Pick a sort order if you want the result alphabetical instead of in its original order.
- Read the counts under the boxes: lines in, lines out, duplicates removed, blank lines removed.
- Copy the cleaned list, or download it as a .txt file.
What it handles
- Spaces and tabs around a line are removed before comparing, so "apple" and " apple " are the same item. Switch trimming off to keep indented lines apart.
- An accented letter matches whichever way it was typed: café written with a combining accent equals the precomposed one.
- Windows line endings survive. A list pasted with CRLF comes back with CRLF, and a single newline at the end stays where it was.
- Repeated blank lines collapse into one. Turn on "Remove blank lines" to drop them all and count them separately.
- Sorting follows the language of the page, so ä sits next to a in German, and item2 comes before item10 instead of after it.
- A 10,000-line list is cleaned as you type. Above 50,000 characters the tool waits a moment after your last keystroke before it recounts.
FAQ
Which copy of a duplicate line is kept?
The first one. Lines two and three of an identical trio are removed and the original keeps its place in the list, so an ordered file stays ordered. The line that survives keeps its own spelling too: with case sensitivity off, "Apple" followed by "APPLE" leaves "Apple". Choose a sort order if the position does not matter to you.
Does uppercase count as a different line?
By default yes, because a list of passwords, codes or file names usually means it. Switch off "Match case" and the comparison ignores case in every alphabet, including Cyrillic and Greek. One exception: the German ß is not folded to ss, so "Straße" and "STRASSE" stay two separate lines.
Is my list uploaded to a server?
No. The comparison runs in your browser, on your device, and no line is sent anywhere. The page keeps your last list in local storage on this device only, so you can come back to it, and Clear deletes that copy. Lists larger than 200 KB are cleaned but not remembered.
How many lines can it handle?
There is no fixed cap. A list of 10,000 lines is cleaned in a few milliseconds, and lists of several megabytes work on a normal phone or laptop. Above 50,000 characters the result waits 80 milliseconds after your last keystroke instead of updating on every letter, which keeps typing smooth.
Can it delete the empty lines as well?
Yes. "Remove blank lines" drops every line that is empty or contains only spaces and tabs, and counts them apart from the duplicates. Leave it off and the blank lines behave like any other line: the first one is kept and the rest are removed as duplicates, which collapses a gap of five empty lines into one.
Does the tool need an internet connection?
Only to load the page. After that the list is compared by the script already in the page, so you can switch off Wi-Fi and keep working. JavaScript has to be enabled; without it the page still shows this text, but the boxes and the counts do not run.
Updated