Remove Duplicate Lines
Paste a list and get it back with the repeats gone, along with a count of how many there were. The first time each line appears is the one kept, so the order you pasted survives.
Runs entirely in your browser. Your files are never uploaded.
How to use Remove Duplicate Lines
- 1Paste your list, one entry per line.
- 2Decide whether capitals and surrounding spaces should count as a difference.
- 3Read how many lines were removed.
- 4Copy the cleaned list.
Which copy is kept
The first appearance of each line stays where it is and every later copy goes. That keeps the order of the original list, which matters when the order means something, like a queue or a log.
Sorting the result is an option if you would rather have it alphabetical. The count of what was removed is the same either way.
Capitals and stray spaces
Two addresses that differ only in capitals are usually the same address, and a line with a trailing space from a spreadsheet paste is usually the same line. Both comparisons are optional so you can decide what counts as a duplicate.
When spaces are ignored for the comparison, the line that gets kept is still the one you pasted, trailing space and all. Turn on trimming if you want the output tidied too.
Showing only the repeats
The opposite view is sometimes the useful one. Switch to showing only duplicated lines and you get a list of the entries that appeared more than once, which is how you find the double bookings or the addresses someone entered twice.
Limits
- Duplicates are whole lines. Two rows that differ in one field are different lines.
- Near-duplicates are not detected: a typo in an address makes it a separate entry.
- Blank lines count as a line and can be removed like any other repeat.
Frequently asked questions
- Does it keep the first or the last copy?
- The first, so the order of your list is preserved. If you need the last of each, reverse the list with the Line Sorter, remove duplicates, then reverse it back.
- Can it find entries that are nearly the same?
- No. Comparison is exact, with optional case and whitespace relaxing. Fuzzy matching needs a tolerance, and a tool that silently merged two slightly different addresses would do more harm than good.
- How many lines can it handle?
- Tens of thousands without trouble, since the work is a single pass with a lookup table. A list long enough to slow your browser down while typing is the practical ceiling.