Find duplicate files by content

Photo libraries, download folders and old backups collect copies of the same file under different names. This script finds them by comparing what is inside the files, not their names. It never deletes anything; it prints groups of identical files and how much space removing the extra copies would free, so you stay in control.

Usage

python3 find_duplicates.py ~/Pictures
python3 find_duplicates.py ~/Downloads ~/Desktop --min-size 1000000   # only files of 1 MB or more
python3 find_duplicates.py ~/Projects --hidden                          # include dot-files and dot-folders

Example output:

3 copies, 293.0 KB each:
  dups/a/big.bin
  dups/b/big-copy.bin
  dups/b/big-copy2.bin

2 copies, 6 B each:
  dups/a/x.txt
  dups/b/y.txt

2 duplicate groups, 585.9 KB could be freed.

The script

#!/usr/bin/env python3
"""List files with identical content under one or more folders. Read-only: nothing is deleted."""
import argparse
import hashlib
import sys
from collections import defaultdict
from pathlib import Path

CHUNK_SIZE = 1024 * 1024


def iter_files(roots, include_hidden):
    seen = set()
    for root in roots:
        for path in root.rglob("*"):
            if not include_hidden and any(part.startswith(".") for part in path.relative_to(root).parts):
                continue
            if path.is_symlink() or not path.is_file():
                continue
            key = path.resolve()
            if key not in seen:
                seen.add(key)
                yield path


def file_hash(path):
    digest = hashlib.sha256()
    with path.open("rb") as f:
        while chunk := f.read(CHUNK_SIZE):
            digest.update(chunk)
    return digest.hexdigest()


def find_duplicates(roots, min_size, include_hidden):
    by_size = defaultdict(list)
    for path in iter_files(roots, include_hidden):
        try:
            size = path.stat().st_size
        except OSError:
            continue
        if size >= min_size:
            by_size[size].append(path)

    groups = []
    for size, paths in by_size.items():
        if len(paths) < 2:
            continue
        by_hash = defaultdict(list)
        for path in paths:
            try:
                by_hash[file_hash(path)].append(path)
            except OSError as exc:
                print(f"skipped {path}: {exc}", file=sys.stderr)
        groups.extend((size, group) for group in by_hash.values() if len(group) > 1)
    return sorted(groups, key=lambda item: item[0] * (len(item[1]) - 1), reverse=True)


def human_size(num_bytes):
    for unit in ("B", "KB", "MB", "GB"):
        if num_bytes < 1024:
            return f"{num_bytes:.0f} {unit}" if unit == "B" else f"{num_bytes:.1f} {unit}"
        num_bytes /= 1024
    return f"{num_bytes:.1f} TB"


def main():
    parser = argparse.ArgumentParser(description="Find files with identical content.")
    parser.add_argument("folders", nargs="+", type=Path, help="folders to scan")
    parser.add_argument("--min-size", type=int, default=1, help="ignore files smaller than this many bytes (default: 1)")
    parser.add_argument("--hidden", action="store_true", help="also scan hidden files and folders")
    args = parser.parse_args()

    for folder in args.folders:
        if not folder.is_dir():
            sys.exit(f"error: {folder} is not a folder")

    groups = find_duplicates(args.folders, args.min_size, args.hidden)
    wasted = 0
    for size, paths in groups:
        wasted += size * (len(paths) - 1)
        print(f"{len(paths)} copies, {human_size(size)} each:")
        for path in sorted(paths):
            print(f"  {path}")
        print()

    print(f"{len(groups)} duplicate groups, {human_size(wasted)} could be freed.")


if __name__ == "__main__":
    main()

How it works

1. Walk the folders. Path.rglob("*") visits every entry below each folder. Symlinks are skipped so the same file is not counted twice through a link, and hidden paths are skipped unless you pass --hidden.
2. Group by size. Files with different sizes cannot be identical, and reading a size from the file system is almost free. Most files are ruled out here without being opened.
3. Hash only the candidates. For sizes shared by two or more files, the script computes a SHA-256 digest. Files are read in 1 MB chunks, so even huge videos never have to fit in memory. The while chunk := f.read(...) loop uses the assignment expression (walrus operator) to read and test in one step.
4. Report. Groups are sorted by wasted space, largest first. Unreadable files are reported on standard error and skipped instead of crashing the scan.

Two different files having the same SHA-256 digest is not a realistic concern for this purpose, so matching digests can be treated as matching content. If you want absolute certainty before deleting, compare a pair byte by byte with filecmp.cmp(a, b, shallow=False).

Acting on the results

Review the list, then delete copies yourself or move them to a holding folder first. If you extend the script to delete automatically, keep a dry-run default, require an explicit flag such as --delete, always keep one file per group, and remember that Path.unlink() is permanent: it does not use the trash.