Recently ran into a problem with a large distributed GlusterFS filesystem. All of a sudden we started getting errors about no free space on device when trying to write files. The individual bricks on the distributed GlusterFS filesystem were hosted on XFS formatted partitions. After some investigation we found that when the size of the brick’s XFS filesystem exceeded 16TB the “no free space” errors started.
The solution was to add “inode64” to the mount options for the XFS partition. The XFS FAQ states the following:
By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like “disk full” when you still have plenty space free, but there’s no more place in the first TB to create a new inode. Also, performance sucks.
To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.
After adding “inode64” to /etc/fstab, we mounted and unmounted the filesystem and restarted Glusterd. Now the distributed partition is working correctly.