How to remove duplicate lines from text online

You merge three newsletter sign-up exports, and Dana from Denver is now on the list four times. She won't enjoy getting four copies of every email. Removing duplicate lines is a small job, but there are a few decisions hiding in it.

Why duplicate lines are a problem

Duplicates turn up in almost any text data. Logs repeat the same error during an outage. Database exports repeat rows after a bad join. Scrapers collect the same URL twice when pagination goes wrong. Either way, duplicates waste space, skew your counts and make data harder to read.

The tricky part is deciding what counts as a duplicate. Should Apple match apple? Should a trailing space matter? The right answer depends on the data, so this guide covers both the options and the reasons to pick them.

Common use cases

Cleaning log files

During an outage, one error can write the same message thousands of times a minute. To see the distinct problems, you need to collapse the repeats. There's a catch, though: log lines usually start with a timestamp, and that makes every line unique:

[2026-02-10 08:15:01] ERROR: Connection refused to database host db-primary:5432
[2026-02-10 08:15:02] ERROR: Connection refused to database host db-primary:5432
[2026-02-10 08:15:03] ERROR: Connection refused to database host db-primary:5432
[2026-02-10 08:15:04] WARN: Retry attempt 1 for query SELECT * FROM users
[2026-02-10 08:15:05] ERROR: Connection refused to database host db-primary:5432
[2026-02-10 08:15:05] WARN: Retry attempt 1 for query SELECT * FROM users

Remove the timestamp and only two distinct messages remain: the database is refusing connections, and the app keeps retrying. On the command line you can drop the first two space-separated fields (the date and time) and count what's left:

cut -d' ' -f3- app.log | sort | uniq -c | sort -rn

      4 ERROR: Connection refused to database host db-primary:5432
      2 WARN: Retry attempt 1 for query SELECT * FROM users

Deduplicating CSV rows

Exports from databases, CRMs and spreadsheets often contain repeated rows, for example a customer imported from two sources. For a one-off clean-up, removing duplicate lines is faster than writing a SQL query or a spreadsheet formula. The header survives because it's the first line:

name,email,plan
Dana Kim,dana@example.com,pro
Luis Ortega,luis@example.com,free
Dana Kim,dana@example.com,pro
Amara Okafor,amara@example.com,pro
Luis Ortega,luis@example.com,free

name,email,plan
Dana Kim,dana@example.com,pro
Luis Ortega,luis@example.com,free
Amara Okafor,amara@example.com,pro

This only catches rows that are identical, character for character. Rows that share an email but differ in another column need deduplication by column, which the awk examples below cover.

Cleaning URL and email lists

Crawls, sitemaps and analytics exports produce long URL lists with repeats, and sending a campaign to a list with duplicate addresses annoys subscribers and hurts deliverability. Case-insensitive matching and whitespace trimming catch most of these. Variants such as a trailing slash or extra query parameters are different strings, so normalize them before you deduplicate.

Merging configuration files

Combining hosts files, dependency lists or allowlists from several sources leaves repeated entries. Removing identical lines tidies them up. It won't resolve conflicts, though: in a merged .env file, PORT=3000 and PORT=8080 are different lines and both stay, so check for repeated keys separately.

Case sensitivity and whitespace

Whether case should matter depends on the data. Email domains are case-insensitive, and while the part before the @ can technically be case-sensitive, almost every mail provider ignores case, so treat Alice@Example.com and alice@example.com as the same address. URL paths are case-sensitive on most servers, so /About and /about may be two different pages.

Whitespace is sneakier, because you can't see it. Lines that look identical can differ by a trailing space, or by a Windows line ending (CRLF) that leaves an invisible \r at the end. Trimming each line before comparing catches these. It won't fix a tab in the middle of a line, though.

Removing duplicates on the command line

On macOS and Linux the classic approach is sort with uniq:

# Sort, then remove duplicates (changes line order)
sort input.txt | uniq > output.txt

# The same in one step
sort -u input.txt > output.txt

# Count how often each line appears, most frequent first
sort input.txt | uniq -c | sort -rn

# Show only the lines that appear more than once
sort input.txt | uniq -d

Why sort first? uniq only removes duplicates that sit next to each other, so sorting puts them side by side. The downside is that you lose the original order. To keep it, use awk:

# Remove duplicates and keep the original order
awk '!seen[$0]++' input.txt > output.txt

# The same, ignoring case
awk '!seen[tolower($0)]++' input.txt > output.txt

# Keep the first row for each value in column 2 of a simple CSV
awk -F',' '!seen[$2]++' input.csv > output.csv

It looks like line noise, but it's simple. seen is a lookup table keyed by the line. The first time a line appears, seen[$0]++ returns 0, the ! turns that into true, and awk prints the line. After that the count is positive, the expression is false and the line is skipped. The column version splits on every comma, so it doesn't understand quoted fields.

In Python, a set does the same job:

seen = set()
with open("input.txt") as f_in, open("output.txt", "w") as f_out:
    for line in f_in:
        key = line.rstrip("\r\n")
        if key not in seen:
            seen.add(key)
            f_out.write(key + "\n")

Handling large files

The awk and Python versions keep every unique line in memory, which becomes a problem for multi-gigabyte files with many distinct lines. Options for large data:

• sort -u sorts with temporary files on disk, so it handles files larger than RAM. It's the most reliable command-line choice, but it changes the order.
• Hashes instead of lines. Store a fixed-size hash of each line, such as SHA-256, instead of the text. At 32 bytes per hash, 100 million unique lines need about 3.2 GB before data structure overhead.
• Bloom filters use a fraction of the memory. They can say for certain that a line is new, but only that a line has probably been seen, so a few unique lines may be dropped. That's fine when approximate results are good enough.
• A database. Insert lines into a SQLite table with a unique constraint using INSERT OR IGNORE. SQLite keeps the data on disk, so size is limited by disk space rather than memory.

For the text you deal with day to day, a few thousand lines or a few megabytes, pasting into a browser tool is quicker than any of these.

How to use the duplicate line remover

1. Open the duplicate line remover and paste your text: plain lines, CSV rows, log output or any other line-based content.
2. Set the options:
   • Case sensitive (on by default). Turn it off to treat Apple and apple as duplicates. The first spelling is kept.
   • Trim whitespace (on by default). Leading and trailing spaces are ignored when comparing and removed from the output.
   • Ignore empty lines. Drops every blank line instead of keeping one.
   • Sort output. Sorts the remaining lines alphabetically instead of keeping the original order.
3. Select Remove duplicates. The first occurrence of each line is kept, and the tool shows the total number of lines, the number of unique lines and how many were removed.
4. Select Copy to copy the result.

Everything runs in your browser, so the text you paste stays on your device. For very large files or automated jobs, use the command-line approaches above.