[Solved] Problems using overlay2 on docker on a VPS

Message

PatomaS · Post by **PatomaS** » Mon Sep 04, 2023 4:04 am

Hi

I'm trying to run docker on a VPS and I'm having some problems with the overlay2 storage.

The setting on the VPS is the same as my testing machine at home where everything works. Kernel options are set the same, daemon.json, etc are configured the same.

I have checked the Gentoo documentation about docker and QEMU a couple of times and I believe I'm not missing anything; this is not the first time that I use either package but is the first time that I use both together.

I have checked documentation in the docker website, posts on stackoverflow and a couple of other places but none seem to address the problem I'm having except for recommending using overlay2 and pygrub if the system is running on Xen, which is not the case as far as I can tell.

This is the error reported by docker
cat /var/log/docker-err.log

Code: Select all

time="2023-09-04T01:24:14.959884415Z" level=info msg="Starting up"
time="2023-09-04T01:24:14.991187440Z" level=info msg="[graphdriver] trying configured driver: overlay2"
time="2023-09-04T01:24:14.992365941Z" level=error msg="failed to mount overlay: no such device" storage-driver=overlay2
failed to start daemon: error initializing graphdriver: driver not supported: overlay2

this is the status when i check services running
rc-status

Code: Select all

rc-status 
Runlevel: default
 net.enp0s3                                                                              [  started  ]
 netmount                                                                                [  started  ]
 metalog                                                                                 [  started  ]
 sshd                                                                                    [  started  ]
 chronyd                                                                                 [  started  ]
 cronie                                                                                  [  started  ]
 docker                                                                                  [  crashed  ]
 local                                                                                   [  started  ]
Dynamic Runlevel: hotplugged
Dynamic Runlevel: needed/wanted
 containerd                                                                              [  started  ]

results of running /usr/share/docker/contrib/check-config.sh

Code: Select all

Info: reading kernel config from /boot/config-6.1.31-gentoo ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled
- CONFIG_NETFILTER_XT_MARK: enabled
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: enabled
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
      - CONFIG_NETFILTER_XT_MATCH_BPF: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled
    - CONFIG_NF_CONNTRACK_FTP: enabled
    - CONFIG_NF_NAT_TFTP: enabled
    - CONFIG_NF_CONNTRACK_TFTP: enabled
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

The only message in dmesg related to overlay
dmesg | grep -i overlay

Code: Select all

[    9.881988] request_module fs-overlay succeeded, but still no fs?

This is the configurations I have

docker -v

Code: Select all

Docker version 24.0.5, build ced0996600

uname -a

Code: Select all

Linux patito 6.1.31-gentoo #3 SMP PREEMPT_DYNAMIC Sun Jun 25 09:04:44 GMT 2023 x86_64 QEMU Virtual CPU version 2.5+ AuthenticAMD GNU/Linux

kernel
cat /usr/src/linux/.config | grep -i overlay

Code: Select all

CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
CONFIG_OVERLAY_FS=y
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

docker - daemon.json
/etc/docker/daemon.json

Code: Select all

{
  "storage-driver": "overlay2"
}

docker - /etc/conf.d/docker
cat /etc/conf.d/docker

Code: Select all

# /etc/conf.d/docker: config file for /etc/init.d/docker

# where the docker daemon output gets piped
# this contains both stdout and stderr. If  you need to separate them,
# see the settings below
DOCKER_LOGFILE="/var/log/docker.log"

# where the docker daemon stdout gets piped
# if this is not set, DOCKER_LOGFILE is used
#DOCKER_OUTFILE="/var/log/docker-out.log"

# where the docker daemon stderr gets piped
# if this is not set, DOCKER_LOGFILE is used
DOCKER_ERRFILE="/var/log/docker-err.log"

# where docker's pid get stored
#DOCKER_PIDFILE="/run/docker.pid"

# Settings for process limits (ulimit)
#DOCKER_ULIMIT="-c unlimited -n 1048576 -u unlimited"

# seconds to wait for sending SIGTERM and SIGKILL signals when stopping docker
#DOCKER_RETRY="TERM/60/KILL/10"

# where the docker daemon itself is run from
#DOCKERD_BINARY="/usr/bin/dockerd"

# any other random options you want to pass to docker
# DOCKER_OPTS="--log-level info --selinux-enabled --data-root /mnt/docker"
DOCKER_OPTS="--log-level info --data-root /mnt/docker"

Any help will be very much appreciated

PatomaS · Post by **PatomaS** » Thu Sep 07, 2023 12:42 am

After posting a reply to aviro in stackexchange, I realized that my kernel version, 6.1.46, was not the same as in the original message, 6.1.31. That could perfectly be because I was copying most of the information from the original site where I posted the question, the docker community forums, and I have done a lot of tests and attempts to fix the problem, but that got me thinking about the kernel version loaded.

Checked that /usr/src/linux was pointing to the right kernel, 6.1.46, and it was, recompiled with overlay fs as a module to post the information related on the original post, did make && make modules_install && make install followed by grub-mkconfig -o /boot/grub/grub.cfg and rebooted.

After rebooting, I ran a uname -a again and the kernel was 6.1.31, why?

May be I did the mkconfig with the wrong path so I did it again, carefully, and rebooted again. Same thing, still 6.1.31.

After that reboot I checked /boot/grub/grub.cfg and the right option, 6.1.46 was there as the first option to boot the system. So just in case there was a problem with some random line not being updated, I did a rm /boot/*6.1.31* and ran grub-mkconfig -o /boot/grub/grub.cfg, checked with grep and there was no trace of 6.1.31, only 6.1.46.

Rebooted again and there it was, 6.1.31 again. But everything looked fine, it should have booted 6.1.46.

Then I rebooted and controled manually the boot process, editing by hand the numbers to load 6.1.46 but the system failed, couldn't find the kernel.

That was key.

I changed back to 6.1.31, let it boot and checked my mount points (/boot is a different partition).

And there it was, or should I say, it wasn't, /boot was not being mounted. At some point on the life of this system, I made a mistake on /etc/fstab and let /boot with noauto so it was not being mounted.

I changed that, removed the noauto for /boot on /etc/fstab, rebooted the system and voila!. The system was 6.1.46 and overlayfs was working.

Although, and this is the new part that I don't get. even when /boot was a folder and not a partition, the 6.1.46 kernel was there and the /boot/grub/grub.cfg had the correct configuration, so why was it not being read and loaded?. On the grub menu presented at boot, there was only one option, 6.1.31 but I couldn't find it on the disk not with locate and not with find, so where is it? how was it being loaded? the /boot/grub/grub.cfg read from the disk with cat had 6.1.46.

Well, after this long story, I have solved the problem and got a new enigma, which, for now is going to remain unsolved because I had used too much time already.

Thanks for the help, without trying to answer that last comment from aviro I wouldn't have stumbled into the solution.

Post by Hu » Thu Sep 07, 2023 2:48 pm

PatomaS wrote:Although, and this is the new part that I don't get. even when /boot was a folder and not a partition, the 6.1.46 kernel was there and the /boot/grub/grub.cfg had the correct configuration, so why was it not being read and loaded?. On the grub menu presented at boot, there was only one option, 6.1.31 but I couldn't find it on the disk not with locate and not with find, so where is it? how was it being loaded? the /boot/grub/grub.cfg read from the disk with cat had 6.1.46.

When you mount a filesystem, you place it over, and hide the contents of, a directory. In this case, you had a filesystem on a device (which I will call /dev/sda1 for convenience, but it might be different for you) which you used as your root filesystem, and another filesystem on a device (which I will call /dev/sda2) which you used as your boot filesystem. Due to your error, you wrote to a directory on /dev/sda1 which happens to be named /boot. You intended to write to the directory / on /dev/sda2. Your bootloader was configured to read from /grub/grub.cfg (note: not a typo, there is no /boot in the filesystem that is normally mounted as /boot on your root filesystem (though some people will create a symlink boot -> . for convenience reasons)) on /dev/sda2, so it ignored any changes you made to /dev/sda1.

It is not clear to me if changing /etc/fstab is the only thing you did to get this working. Your post reads like it is, but I cannot see how that would be right. /etc/fstab is only relevant after mounting root, so it cannot influence what kernel the bootloader offers you. It can influence whether you correctly install a new kernel, and whether you correctly register that newly installed kernel with the bootloader. If you fixed /etc/fstab, then reran the kernel install + bootloader configuration steps, that would explain your success.