openrc failed to start docker

Message

papas · Post by **papas** » Sun May 17, 2020 11:10 am

Hello guys, i don't know when and how this started, but i was successful install and running docker for a long time without any problem. After a period of some days (3-7) without use of docker, today i tried to start it, as usual openrc prompted for the root password and everything seems to be ok:

Code: Select all

  master@gentoo ~ $ sudo rc-service docker start
Password: 
Authenticating root.
Password: 
 * Starting docker ...                                                                                                                                                                            [ ok ]

but there is no docker running:

Code: Select all

master@gentoo ~ $ sudo rc-service docker status
Authenticating root.
Password: 
 * status: crashed

with no error at all. Starting docker in debug mode I am getting this messages:

Code: Select all

master@gentoo ~ $ sudo /etc/init.d/docker -d start
Authenticating root.
Password: 
+ sourcex -e /etc/rc.conf
+ '[' -e = -e ']'
+ shift
+ '[' -e /etc/rc.conf ']'
+ . /etc/rc.conf
++ rc_shell=/sbin/sulogin
++ rc_logger=YES
++ unicode=YES
++ rc_tty_number=12
+ '[' -d /etc/rc.conf.d ']'
+ _conf_d=/etc/init.d/../conf.d
+ _c=docker
+ '[' -n docker -a docker '!=' docker ']'
+ unset _c
+ sourcex -e /etc/init.d/../conf.d/docker.default
+ '[' -e = -e ']'
+ shift
+ '[' -e /etc/init.d/../conf.d/docker.default ']'
+ return 1
+ sourcex -e /etc/init.d/../conf.d/docker
+ '[' -e = -e ']'
+ shift
+ '[' -e /etc/init.d/../conf.d/docker ']'
+ . /etc/init.d/../conf.d/docker
++ DOCKER_LOGFILE=/var/log/docker.log
++ DOCKER_PIDFILE=/run/docker.pid
++ DOCKERD_BINARY=/usr/bin/dockerd
+ unset _conf_d
+ sourcex /lib/rc/sh/runit.sh
+ '[' /lib/rc/sh/runit.sh = -e ']'
+ . /lib/rc/sh/runit.sh
+ sourcex /lib/rc/sh/s6.sh
+ '[' /lib/rc/sh/s6.sh = -e ']'
+ . /lib/rc/sh/s6.sh
++ '[' -z '' ']'
++ s6_service_path=/var/svc.d/docker
+ sourcex /lib/rc/sh/start-stop-daemon.sh
+ '[' /lib/rc/sh/start-stop-daemon.sh = -e ']'
+ . /lib/rc/sh/start-stop-daemon.sh
+ sourcex /lib/rc/sh/supervise-daemon.sh
+ '[' /lib/rc/sh/supervise-daemon.sh = -e ']'
+ . /lib/rc/sh/supervise-daemon.sh
++ extra_commands='healthcheck unhealthy '
+ sourcex /etc/init.d/docker
+ '[' /etc/init.d/docker = -e ']'
+ . /etc/init.d/docker
++ command=/usr/bin/dockerd
++ pidfile=/run/docker.pid
++ command_args='-p "/run/docker.pid" '
++ DOCKER_LOGFILE=/var/log/docker.log
++ DOCKER_ERRFILE=/var/log/docker.log
++ DOCKER_OUTFILE=/var/log/docker.log
++ start_stop_daemon_args='--background         --stderr "/var/log/docker.log" --stdout "/var/log/docker.log"'
++ extra_started_commands=reload
++ rc_ulimit='-c unlimited -n 1048576 -u unlimited'
++ retry=TERM/60/KILL/10
+ yesno ''
+ '[' -z '' ']'
+ return 1
+ for _cmd in "$@"
+ '[' start '!=' status -a start '!=' describe ']'
+ '[' -n '-c unlimited -n 1048576 -u unlimited' ']'
+ ulimit -c unlimited -n 1048576 -u unlimited
++ command -v cgroup_add_service
+ '[' cgroup_add_service = cgroup_add_service ']'
+ grep -qs /sys/fs/cgroup /proc/1/mountinfo
+ '[' -d /sys/fs/cgroup -a '!' -w /sys/fs/cgroup ']'
+ cgroup_add_service
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/blkio/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/cpu/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/cpuacct/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/cpuset/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/devices/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/freezer/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/hugetlb/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/memory/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/net_cls/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/net_prio/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/openrc/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/perf_event/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/pids/tasks ']'
+ printf %d 0
+ for d in /sys/fs/cgroup/*
+ '[' -w /sys/fs/cgroup/unified/tasks ']'
+ openrc_cgroup=/sys/fs/cgroup/openrc
+ '[' -d /sys/fs/cgroup/openrc ']'
+ cgroup=/sys/fs/cgroup/openrc/docker
+ mkdir -p /sys/fs/cgroup/openrc/docker
+ '[' -w /sys/fs/cgroup/openrc/docker/tasks ']'
+ printf %d 0
++ command -v cgroup_set_limits
+ '[' cgroup_set_limits = cgroup_set_limits ']'
+ cgroup_set_limits
+ local blkio=
+ '[' -n '' ']'
+ local cpu=
+ '[' -n '' ']'
+ local cpuacct=
+ '[' -n '' ']'
+ local cpuset=
+ '[' -n '' ']'
+ local devices=
+ '[' -n '' ']'
+ local hugetlb=
+ '[' -n '' ']'
+ local memory=
+ '[' -n '' ']'
+ local net_cls=
+ '[' -n '' ']'
+ local net_prio=
+ '[' -n '' ']'
+ local pids=
+ '[' -n '' ']'
+ return 0
++ command -v cgroup2_set_limits
+ '[' cgroup2_set_limits = cgroup2_set_limits ']'
+ '[' start = start ']'
+ cgroup2_set_limits
+ local cgroup_path
++ cgroup2_find_path
++ grep -qw cgroup2 /proc/filesystems
++ case "${rc_cgroup_mode:-hybrid}" in
++ printf /sys/fs/cgroup/unified
++ return 0
+ cgroup_path=/sys/fs/cgroup/unified
+ '[' -d /sys/fs/cgroup/unified ']'
+ rc_cgroup_path=/sys/fs/cgroup/unified/docker
+ '[' '!' -d /sys/fs/cgroup/unified/docker ']'
+ mkdir /sys/fs/cgroup/unified/docker
+ '[' -f /sys/fs/cgroup/unified/docker/cgroup.procs ']'
+ printf 0
+ '[' -z '' ']'
+ return 0
+ break
+ eval 'printf '\''%s\n'\'' '
++ printf '%s\n'
+ read _d
+ '[' -n '' ']'
+ read _d
+ '[' 0 -ne 0 ']'
+ unset _d
+ eval 'printf '\''%s\n'\'' '
++ printf '%s\n'
+ read _f
+ '[' -n '' ']'
+ read _f
+ '[' 0 -ne 0 ']'
+ unset _f
+ '[' -n '' ']'
+ '[' -n start ']'
+ '[' start = depend ']'
+ for _cmd in describe start stop status ${extra_commands:-$opts} $extra_started_commands $extra_stopped_commands
+ '[' describe = start ']'
+ for _cmd in describe start stop status ${extra_commands:-$opts} $extra_started_commands $extra_stopped_commands
+ '[' start = start ']'
++ command -v start
+ '[' start = start ']'
+ yesno
+ '[' -z '' ']'
+ return 1
+ for _cmd in $extra_started_commands
+ '[' reload = start ']'
+ for _cmd in $extra_stopped_commands
+ '[' cgroup_cleanup = start ']'
+ unset _cmd
+ case $1 in
+ verify_boot
+ '[' '!' -e /run/openrc/softlevel ']'
+ return 0
++ command -v start_pre
+ '[' start_pre = start_pre ']'
+ start_pre
+ checkpath -f -m 0644 -o root:docker /var/log/docker.log
+ start
+ default_start
+ local func=ssd_start
+ case "$supervisor" in
+ ssd_start
+ '[' -z /usr/bin/dockerd ']'
+ local _background=
+ ebegin 'Starting docker'
 * Starting docker ...
+ yesno ''
+ '[' -z '' ']'
+ return 1
+ yesno ''
+ '[' -z '' ']'
+ return 1
+ '[' -n '' ']'
+ '[' -n '' ']'
+ eval start-stop-daemon --start --exec /usr/bin/dockerd --pidfile /run/docker.pid --background --stderr '"/var/log/docker.log"' --stdout '"/var/log/docker.log"' -- -p '"/run/docker.pid"'
++ start-stop-daemon --start --exec /usr/bin/dockerd --pidfile /run/docker.pid --background --stderr /var/log/docker.log --stdout /var/log/docker.log -- -p /run/docker.pid
+ eend 0 'Failed to start docker'                                                                                                                                                                                                   [ ok ]
+ service_set_value command /usr/bin/dockerd
+ '[' -n '' ']'
+ '[' -n /run/docker.pid ']'
+ service_set_value pidfile /run/docker.pid
+ '[' -n '' ']'
+ return 0
++ command -v start_post
+ '[' '' = start_post ']'
++ command -v cgroup_cleanup
+ '[' cgroup_cleanup = cgroup_cleanup ']'
+ '[' start = stop ']'
++ command -v cgroup2_remove
+ '[' cgroup2_remove = cgroup2_remove ']'
+ '[' start = stop ']'
+ '[' -z /usr/bin/dockerd ']'
+ shift
+ continue 2
+ '[' -n '' ']'
+ exit 0

btw, by running the daemon without the use of openrc docker (dockerd) working fine, what can be wrong?

alamahant · Post by **alamahant** » Sun May 17, 2020 1:15 pm

Please in /etc/rc.conf make sure you have this entry:

Code: Select all

rc_cgroup_mode="legacy"

And reboot.
This is supposed to be a requirement for running lxd in openrc.
So I suppose this requirement extends to docker also.
.........Possibly...
Please see this also
https://wiki.gentoo.org/wiki/LXD#Troubleshooting

papas · Post by **papas** » Sun May 17, 2020 2:04 pm

thank you but it is not working. openrc has a very strange behavior while trying to start docker asking for root password even if i am already logged in as root.

alamahant · Post by **alamahant** » Sun May 17, 2020 2:34 pm

This is not supposed to be happening.
If you are root you are supposed to be able to do anything-no questions asked- unless maybe you have selinux installed etc...
Maybe something deeper is going on in your system...
Is docker maybe built with "hardened" or apparmor" USE flag?
What if you re-emerged docker?

papas · Post by **papas** » Sun May 17, 2020 2:53 pm

i do have selinux but is in permissive mode and the problem is only with docker everything working fine. There is something wrong with rc-service command, if i run

Code: Select all

master@gentoo ~ $ sudo openrc-run /etc/init.d/docker start
 * Caching service dependencies ...                                                                                                                             [ ok ]
 * Starting docker ...                                                                                                                                                      [ ok]

docker run without problems, strange.
i have already re-emerge docker and openrc and i have try with 5.4.38 kernel.

alamahant · Post by **alamahant** » Sun May 17, 2020 3:00 pm

then maybe

Code: Select all

/sbin/rc-service

is selinux-mislabeled.
Maybe if you did a system relabel.
Does

Code: Select all

touch /.autorelabel
reboot

work in Gentoo?

By the way on a different note--I was curious what selinux flags you have in make.conf?
Does your system function in "Enforcing" mode or you get a million errors and violations?
I tried it once but it was inoperable...
Ah yes and I now remember that it used to ask for password when interacting with services and daemons....
If it is working for you maybe you can share it with the community............

papas · Post by **papas** » Sun May 17, 2020 6:00 pm

alamahant wrote:then maybe

By the way on a different note--I was curious what selinux flags you have in make.conf?
Does your system function in "Enforcing" mode or you get a million errors and violations?
I tried it once but it was inoperable...
Ah yes and I now remember that it used to ask for password when interacting with services and daemons....
If it is working for you maybe you can share it with the community............

I maked my own profile:

Code: Select all

master@gentoo ~ $ eselect profile list
Available profile symlink targets:
  [1]   default/linux/amd64/17.0 (stable)
  [2]   default/linux/amd64/17.0/selinux (stable)
  [3]   default/linux/amd64/17.0/hardened (stable)
  [4]   default/linux/amd64/17.0/hardened/selinux (stable)
  [5]   default/linux/amd64/17.0/desktop (stable)
  [6]   default/linux/amd64/17.0/desktop/gnome (stable)
  [7]   default/linux/amd64/17.0/desktop/gnome/systemd (stable)
  [8]   default/linux/amd64/17.0/desktop/plasma (stable)
  [9]   default/linux/amd64/17.0/desktop/plasma/systemd (stable)
  [10]  default/linux/amd64/17.0/developer (stable)
  [11]  default/linux/amd64/17.0/no-multilib (stable)
  [12]  default/linux/amd64/17.0/no-multilib/hardened (stable)
  [13]  default/linux/amd64/17.0/no-multilib/hardened/selinux (stable)
  [14]  default/linux/amd64/17.0/systemd (stable)
  [15]  default/linux/amd64/17.0/x32 (dev)
  [16]  default/linux/amd64/17.1 (stable)
  [17]  default/linux/amd64/17.1/selinux (stable)
  [18]  default/linux/amd64/17.1/hardened (stable)
  [19]  default/linux/amd64/17.1/hardened/selinux (stable)
  [20]  default/linux/amd64/17.1/desktop (stable)
  [21]  default/linux/amd64/17.1/desktop/gnome (stable)
  [22]  default/linux/amd64/17.1/desktop/gnome/systemd (stable)
  [23]  default/linux/amd64/17.1/desktop/plasma (stable)
  [24]  default/linux/amd64/17.1/desktop/plasma/systemd (stable)
  [25]  default/linux/amd64/17.1/developer (stable)
  [26]  default/linux/amd64/17.1/no-multilib (stable)
  [27]  default/linux/amd64/17.1/no-multilib/hardened (stable)
  [28]  default/linux/amd64/17.1/no-multilib/hardened/selinux (stable)
  [29]  default/linux/amd64/17.1/systemd (stable)
  [30]  default/linux/amd64/17.0/musl (exp)
  [31]  default/linux/amd64/17.0/musl/hardened (exp)
  [32]  default/linux/amd64/17.0/musl/hardened/selinux (exp)
  [33]  default/linux/amd64/17.0/uclibc (exp)
  [34]  default/linux/amd64/17.0/uclibc/hardened (exp)
  [35]  local:default/linux/amd64/17.1/desktop/plasma/selinux (exp) *

i followed the wiki and enable selinux. Almost always i am in permissive mode, the last 6 months i emailed the gentoo selinux developer about the support of elogind in selinux and he guided me how to do it. I run elogind under selinux succefully and recently changed from strict to mcs type. I can make policy with audit2allow and run with no problems my kde enviroment, but it is a huge policy. I am facing seriusly problems with some of policies. Unfortunatelly if i try to write a guide, because of my bad English, Gentooers they will sue me.

Code: Select all

master@gentoo ~ $ cat /etc/portage/make.conf 
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CHOST="x86_64-pc-linux-gnu"
#CFLAGS="-march=broadwell -O2 -pipe"
CFLAGS="-O2 -pipe -march=native"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j11"
# NOTE: This stage was built with the bindist Use flag enabled
PORTDIR="/usr/portage"
DISTDIR="/usr/portage/distfiles"
PKGDIR="/usr/portage/packages"
USE=" X experimental alsa ipv6 snmp -consolekit elogind"
# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C
INPUT_DEVICES="libinput evdev synaptics"
VIDEO_CARDS="nvidia"
GENTOO_MIRRORS="ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo http://ftp.ntua.gr/pub/linux/gentoo/ ftp://ftp.ntua.gr/pub/linux/gentoo/"
GRUB_PLATFORMS="efi-64"
ACCEPT_LICENSE="* -@EULA"
POLICY_TYPES="strict mcs mls"

n3bul4 · Post by **n3bul4** » Wed Apr 28, 2021 10:03 pm

I was actually having the same problem running a hardened selinux profile:

default/linux/amd64/17.1/hardened/selinux

Starting docker by using /usr/bin/dockerd in bash works for me.

Taking a closer look at the /etc/init.d/docker script and using the arguments (also from /etc/conf.d/docker) to create a command, I can even run it from bash like this:

start-stop-daemon --background --stderr /var/log/docker-err.log --stdout /var/log/docker-out.log -- /usr/bin/dockerd -p /run/docker.pid --selinux-enabled --storage-driver overlay2

The /etc/init.d/docker script contains the following line:

rc_ulimit="${DOCKER_ULIMIT:--c unlimited -n 1048576 -u unlimited}"

By commenting the above line out, docker can suddenly be started with the /etc/init.d/docker script.
Looks like openrc has problems setting ulimits, but I have currently no idea why this is the case.

Does anybody have an idea?
Would be nice to fix this. As far as I know those ulimits make sense, so it would be nice to have them set.