My gentoo root (/) space is getting full due to docker

chandan.kashyap · n00b Joined: 03 Feb 2018 Posts: 2

We have a gentoo server on which docker Vs 1.11 is running and from past few days / (root) volume is getting full in regular basis and due to this Inode is also getting full which is causing other application down. Please suggest me on following questions:

1. How to remove inode??
2. Why /var/lib/docker/devicemapper is getting alot of storage. However on same configuration other gentoo is working well?
3. Is there any specific command available to release storage ?

My docker and machine info is below :-
docker info

Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.11.0
Storage Driver: devicemapper
Pool Name: docker-8:4-265450-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 5.829 GB
Data Space Total: 107.4 GB
Data Space Available: 3.752 GB
Metadata Space Used: 7.508 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.14 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93 (2015-01-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.21-gentoo
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 3
Total Memory: 2.935 GiB
Name: valor-app-6
ID: 3PRF:EPCR:CIT5:WJCZ:WVOJ:OGGA:X6MA:BRLT:L2YE:7KUO:XT6G:RAHJ
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

szatox · Advocate Joined: 27 Aug 2013 Posts: 3131

chandan.kashyap · n00b Joined: 03 Feb 2018 Posts: 2

Thank You for the help.

Yes, Gentoo is a host server on which docker is running and under docker 2 containers are operational.

Since, 2 Months we are continuously facing issue of storage on the host Gentoo server.As per our investigation, the most used data directories are /var/lib/docker/metadata/devicemapper and usr/mnt/portage

We are not able to understand why this is happening. Althrough, We have some other Gentoo server as well on which docker is running quite well without any issue.

Please suggest us with most appropriate way to rectify the exact cause like Why metadata is consuming alot of storage and also INODE are showing 86% full.

Note: It is our production server and any wrong step will make my server down.

Your quick reply would be highly appreciated.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3131

So my previous post regarding maintaining gentoo holds.
Don't worrt about /usr/portage too much, your system won't die instantly if you break it. It's only used for managing your installed software. You can keep working without it, but you can't change anything. It is also fully recoverable, so even breaking it beyond repair is not such a big deal.

Now, every file consumes an inode from filesystem. Every directory also is a file so it consumes an inode. Every device node or fifo or a socket is a file and consumes an inode.
Many small files consume many inodes (e.g. portage: ~170k in my case). They don't consume much disk space, but if you run out of inodes your disk becomes full.
I'm not a docker guy, but since you use device mapper driver it is possible that you simply have a lot of changes in your image. This actually goes in line with running out of inodes. Every inode modified inside that image inflates volume's metadata. It also goes in line with application not being designed for running inside docker. It may be creating a lot of event files that should be consumed by another process, archived, or simply rotated by a maintenance job. Maybe it's poorly made logging or an interface meant for integration between some complex systems.

Also, how big filesystem you have there? Gentoo requires quite a bit of space for its toolchain and cache. It running out of space could mean you simply need a few GBs more.
Yes, /usr/portage is a significant contributor. The more software you install, the more sources you/re gonna find in /usr/portage/distfiles. Again: eclean helps.