Diagnosing Inode Usage

I recently had an issue whereby one of my Docker servers was complaining that it had run out of space, even though the outputs of df and pydf showed that I clearly had quite a bit of storage free. It turns out that I had run out of inodes instead.

Apparently, the ext4 defaults to providing one inode per 16K bytes of storage.

Checking Filesystem Total Inode Usage

To quickly check if your system is running out of inodes, just run the following command:

df --inodes

After having moved the docker storage location to a larger disk that I added to the server (/dev/sdb1), then this output the following for me:

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
udev            496728     348  496380    1% /dev
tmpfs           501618     581  501037    1% /run
/dev/sda1      1302528   80375 1222153    7% /
tmpfs           501618      34  501584    1% /dev/shm
tmpfs           501618       2  501616    1% /run/lock
/dev/sdb1      1966080 1268154  697926   65% /mnt/disk1
/dev/sda15           0       0       0     - /boot/efi
tmpfs           100323      20  100303    1% /run/user/1000

You can see that my inode percentage is at 65% for that filesystem. However the normal output of df shows:

Filesystem     1K-blocks    Used Available Use% Mounted on
udev             1986912       0   1986912   0% /dev
tmpfs             401296     752    400544   1% /run
/dev/sda1       20470152 5136616  14463900  27% /
tmpfs            2006472      88   2006384   1% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
/dev/sdb1       30786468 4761876  24435396  17% /mnt/disk1
/dev/sda15        126678   10922    115756   9% /boot/efi
tmpfs             401292       0    401292   0% /run/user/1000

... which shows only 17% disk usage. This demonstrates just how much the inodes percentage usage can be way out of alignment with general disk usage.

Inode Usage Breakdown

After determining if your filesystem is running out of inodes, you likely want to figure out where they are all being used up. Unfortunately, I couldn't find a nice tool similar to ncdu for this, so for now I'm using the following command to determine the inode usage in whatever directory I'm currently in (you may wish to be the root user to prevent hitting permission issues with accessing directories).

find . \
  -type f \
  | cut -d/ -f2 \
  | \sort \
  | uniq --count \
  | sort --numeric-sort \
  | column --table

I got this output when I ran it on my local /var/lib/docker directory (Docker's default storage location):

1        engine-id
1        network
21       containers
245      buildkit
11502    volumes
12431    image
2085208  overlay2

You can use this to quickly determine which directories have the largest amount of inodes, and drill down to see if there is anything you can quickly delete to free some up.

You could have the largest number at the top by adding --reverse to the sort line in the command, but I like having the largest number at the bottom to prevent having to scroll up, as I only really care about the larger directories.