Find Large Files on Linux: du, find and ls Disagree

Linux Shell Prompt

Two different questions, two different commands. For “what is eating my disk”, walk the tree by allocated space and let it roll up by directory: du -xk /path | sort -nr | head -20. For “which individual files are big”, ask find: find /path -type f -size +100M. They disagree, sometimes by four orders of magnitude, and the rest of this page is the measured proof of when and why. Once you know which file it is, the ENOSPC page covers the cases where deleting it does not give the space back.

du -a lists directories, so the top of your list is not files

The previous version of this page opened with du -a . | sort -n -r | head -n 10 described as “a quick way to find the biggest files”. du -a reports every file and every directory. On a tree containing one 8 MiB file and one 4 MiB file:

$ du -a tree | sort -n -r | head -n 10
24584	tree
16384	tree/sub/big1.bin
16384	tree/sub
8192	tree/other/big2.bin
8192	tree/other
8	tree/small.txt

Every real file appears twice — once as itself, once as the parent directory that contains it — and the largest entry is the root of the search, which is not a file at all. On a real filesystem with deep directories, a head -10 of that output is almost entirely directories. If you want the directory rollup, that is the right tool and you should drop -a: gdu -xk --max-depth=1 on GNU, du -xkd1 on BSD — the long option is GNU-only and both spellings were run here. If you want files, filter with find instead.

The numbers are in different units on BSD and GNU

Same file, same tree, two du implementations on the same machine:

$ du -a tree | sort -nr | head -3      # BSD du
24584	tree
16384	tree/sub/big1.bin
16384	tree/sub

$ gdu -a tree | sort -nr | head -3     # GNU coreutils 9.11
12292	tree
8192	tree/sub/big1.bin
8192	tree/sub

BSD du counts 512-byte blocks, GNU counts 1024-byte blocks, and nothing in the output says which. A number you paste from a macOS terminal into a Linux runbook is off by a factor of two. Two ways out, both verified here: pass -k and both print 7168 for the same 7 MiB file, or pass -h and let each print 7.0M. The direction reverses under GNU’s POSIX mode — POSIXLY_CORRECT=1 gdu -a printed 14336 where plain gdu -a printed 7168 — so -k is the only spelling that is stable in both directions.

A sparse file is huge to find and tiny to du

This is the trap that makes “find the biggest file and delete it” fail to free anything. Create a file with a hole in it:

$ dd if=/dev/zero of=sparse.bin bs=1 count=1 seek=100m
$ ls -l sparse.bin
-rw-r--r--  1 …  104857601 …  sparse.bin
$ stat -f '%z bytes  %b blocks' sparse.bin
104857601 bytes  8 blocks
$ du -k sparse.bin
4	sparse.bin
$ find . -type f -size +50M
./sparse.bin

100 MB by apparent size, 4 KiB actually allocated. find -size, ls -l and gdu --apparent-size -k (102401) all report the big number; du reports the small one, and du is the one that matches what you get back by deleting it. Sparse files are not exotic — VM disk images, database files, core dumps and downloaded torrents are routinely sparse. Whenever the two tools disagree by orders of magnitude, this is usually why.

Hard links go wrong in both directions

One 5 MiB file with two names in the same directory:

$ du -a hl
10240	hl/original.bin
10240	hl

$ du -al hl
10240	hl/original.bin
10240	hl/hardlink.bin
20480	hl

$ find hl -type f -exec ls -l {} \; | awk '{print $5, $9}'
5242880 hl/original.bin
5242880 hl/hardlink.bin

$ du -sh hl
5.0M	hl

Default du counts the inode once and omits the second name from the listing entirely — hunting for hardlink.bin in that output finds nothing. Add -l and it appears, but the total doubles to 20480 blocks for 5 MiB of real data. find lists both at full size, so a find-based audit of a tree full of hard links (a package cache, a deduplicating backup, a node_modules store) overstates it. Neither is a bug; they answer different questions. The one that matches free space is plain du.

Two of the old page’s five examples do not do what it says

ls -lSh | head was offered as “a fast summary” of the biggest files. It is not recursive, and it sizes directories by their directory entry rather than their contents. In the tree above, whose biggest file is 8 MiB:

$ ls -lSh tree | head
total 8
drwxr-xr-x@ 3 …    96B … other
drwxr-xr-x@ 3 …    96B … sub
-rw-r--r--@ 1 …     5B … small.txt

The 8 MiB file is not in the output at all. ls -S is fine for one flat directory of files and useless as a disk-space tool.

find / -type f -size +50M -exec ls -lh {} \; | awk '{ printf "%s %s\n", $5, $9 }' loses filenames. awk splits on whitespace, so $9 is the first word of the name:

6.0M t/my
6.0M t/plain.bin

The real name is t/my big video.mov. It also forks ls once per match — over five files, -exec … \; ran five times where -exec … {} + ran once. Ask find for the size directly and the problem disappears: -exec stat -c '%s %n' {} + with GNU stat, -exec stat -f '%z %N' {} + with BSD stat. Both were run here against the same tree and both printed 6291456 t/my big video.mov intact.

Two things the old page got right: find /home -type f -size +100M is the correct shape for a size hunt, and ncdu is a good interactive browser (not installed here, so that is a recommendation, not a demonstration). Beware -size‘s units — a bare number means 512-byte blocks, and BSD and GNU round differently; the find tutorial has that measured. Add -xdev to a system-wide sweep, and send stderr to a file rather than /dev/null so you can tell “nothing there” from “could not look”.

The script that reproduces every block above builds its own scratch tree, including the sparse file and the hard links, and removes it on exit.

Photo of author
Sudhir P. founded HeatWare.com in 1999 and has built and operated it full-stack ever since; it is now used by more than 88,000 people. He writes here about the PostgreSQL, MySQL, Linux and DevOps work that keeps it running. Articles are rewritten only after the commands in them have actually been run, and the verification scripts are published alongside them so anyone can check the claims. Reach him at blog@heatware.net.