Remove Duplicate Lines

How it works

The tool splits your input on line breaks, then walks the lines in order and builds a map keyed by the normalised form of each line. That key is what decides whether two lines are "the same": with every option off it is the raw line, with trim on it is the line without leading and trailing whitespace, and with ignore case on it is also lower-cased. The first line that produces a given key is stored, and every later line that produces the same key is dropped — unless you selected keep last, in which case the stored value is overwritten so the final occurrence wins and the earlier copies fall away.

Once duplicates are gone, the surviving lines are ordered according to your sorting choice. A–Z and Z–A use a locale-aware comparison, and natural order uses Intl.Collator with the numeric option so that item2 sorts before item10 rather than after item1. Keep original order skips sorting entirely and leaves the survivors in the sequence they appeared in, which is the only mode that is safe for playlists, routes and step-by-step instructions.

What "duplicate" means here

A duplicate is an exact match after the normalisation you chose — nothing more. The comparison is on the whole line, so Apple Inc and Apple are different values no matter which options are set, and the tool never tries to guess at near-matches, fuzzy spelling or word-level overlap.

Case is the clearest example. Apple and apple are two distinct lines while ignore case is off, and one value as soon as you switch it on. Whitespace behaves the same way: with trim off, "banana" and "banana  " are different lines; with trim on they collapse into one. That also means ignore case and trim change the text you get back — the surviving line keeps its original spelling and spacing from the input, so deduplicating APPLE followed by apple with keep first returns APPLE, while keep last returns apple.

Empty lines get their own switch. With ignore empty lines off, every blank line is a value, so several blank lines in a row collapse to one; with it on, blank lines are discarded entirely and never appear in the output or in the counts.

How to use it

1. Paste or type your list into the input box, one item per line.
2. Set the normalisation options. Leave them all off if you want a strictly literal comparison, or turn on trim first — it is the one that saves the most surprises.
3. Choose whether the first or the last occurrence of each value should survive, then pick an output order.
4. Watch the four counters: input lines, output lines, duplicates removed and unique values. They update as you type, so you can see the effect of each switch immediately.
5. Copy the result or download it as a plain .txt file. The download is produced locally with a Blob, so there is no upload and no round trip.

Two honest limits are worth knowing. The counters count lines, not words, and a line that differs only by invisible characters — a non-breaking space, a zero-width space, a stray carriage return from a Windows clipboard — counts as a distinct value even with trim on, because trim only removes the standard whitespace at the two ends. If a list refuses to shrink and you cannot see why, the cause is usually invisible characters or trailing punctuation rather than a bug in the comparison.

Very large inputs are handled in memory on the main thread. Tens of thousands of lines are comfortable; several hundred thousand will make the first keystroke pause for a moment while the comparison runs. Nothing is truncated, but the page may feel briefly unresponsive on very large pastes.