[SOLVED] unprivileged lxc container

Message

skunk · Post by **skunk** » Fri Feb 19, 2016 4:21 pm

hi!
i'm trying to deploy unprivileged lxc containers by following this guide with this error:

mybp@rock ~ $ lxc-create -t download -n test -f ~/.config/lxc/mybp.conf -- -d ubuntu -r xenial -a amd64
WARN: could not reopen tty: Permission denied
unshare: Operation not permitted
read pipe: Success
lxc_container: lxccontainer.c: do_create_container_dir: 760 Failed to chown container dir
lxc_container: lxc_create.c: main: 271 Error creating container test

searching the web i've found out both /sys/fs/cgroup/cpuset/cgroup.clone_children and /proc/sys/kernel/unprivileged_userns_clone should be set to 1.
while the former was already set, the latter doesn't even exist nor does a kernel.unprivileged_userns_clone sysctl variable...
is there any kernel module i'm missing? which one?

thank you

alinefr · Post by **alinefr** » Fri Feb 19, 2016 4:43 pm

As I understand, unprivileged does not mean that you could create it with an ordinary user. You still should build it with root privileges (and the examples in the wiki are executed by root).

skunk · Post by **skunk** » Fri Feb 19, 2016 5:17 pm

i guessed it was a typo since the author even runs lxc-start as root...
however even creating the container as root fails:

Code: Select all

rock ~ # lxc-create -t download -n mybp -f /home/mybp/.config/lxc/mybp.conf -- -d ubuntu -r xenial -a amd64
newuidmap: uid range [0-65536) -> [100000-165536) not allowed
error mapping child
setgid: Invalid argument
lxc_container: lxccontainer.c: create_run_template: 1125 container creation template for mybp failed
lxc_container: lxc_create.c: main: 271 Error creating container mybp

rock ~ # cat /home/mybp/.config/lxc/mybp.conf
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = eth0
lxc.network.name = eth0
lxc.network.ipv4 = 192.168.10.100/24
lxc.network.ipv4.gateway = 192.168.10.1
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536


rock ~ # grep mybp /etc/sub* 2>/dev/null
/etc/subgid:mybp:100000:65536
/etc/subuid:mybp:100000:65536

as a side note: manually deployed privileged containers works fine...

skunk · Post by **skunk** » Fri Feb 19, 2016 9:25 pm

it turns out hardened-sources is the culpit, so i've to figure out which of these options should be turned on/off:

Code: Select all

CONFIG_GRACE_PERIOD=m
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CONFIG_AUTO=y
# CONFIG_GRKERNSEC_CONFIG_CUSTOM is not set
CONFIG_GRKERNSEC_CONFIG_SERVER=y
# CONFIG_GRKERNSEC_CONFIG_DESKTOP is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_NONE=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_GUEST is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_HOST is not set
CONFIG_GRKERNSEC_CONFIG_PRIORITY_PERF=y
# CONFIG_GRKERNSEC_CONFIG_PRIORITY_SECURITY is not set
CONFIG_GRKERNSEC_PROC_GID=10
CONFIG_GRKERNSEC_SYMLINKOWN_GID=81
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_BPF_HARDEN=y
CONFIG_GRKERNSEC_PERF_HARDEN=y
CONFIG_GRKERNSEC_RAND_THREADSTACK=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_KSTACKOVERFLOW=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODHARDEN=y
CONFIG_GRKERNSEC_HIDESYM=y
CONFIG_GRKERNSEC_RANDSTRUCT=y
CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y
CONFIG_GRKERNSEC_KERN_LOCKOUT=y
# CONFIG_GRKERNSEC_NO_RBAC is not set
# CONFIG_GRKERNSEC_ACL_HIDEKERN is not set
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
# CONFIG_GRKERNSEC_PROC is not set
CONFIG_GRKERNSEC_LINK=y
# CONFIG_GRKERNSEC_SYMLINKOWN is not set
CONFIG_GRKERNSEC_FIFO=y
# CONFIG_GRKERNSEC_SYSFS_RESTRICT is not set
# CONFIG_GRKERNSEC_ROFS is not set
CONFIG_GRKERNSEC_DEVICE_SIDECHANNEL=y
CONFIG_GRKERNSEC_CHROOT=y
# CONFIG_GRKERNSEC_CHROOT_MOUNT is not set
# CONFIG_GRKERNSEC_CHROOT_DOUBLE is not set
# CONFIG_GRKERNSEC_CHROOT_PIVOT is not set
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
# CONFIG_GRKERNSEC_CHROOT_CHMOD is not set
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_CHROOT_RENAME=y
# CONFIG_GRKERNSEC_CHROOT_CAPS is not set
# CONFIG_GRKERNSEC_AUDIT_GROUP is not set
# CONFIG_GRKERNSEC_EXECLOG is not set
CONFIG_GRKERNSEC_RESLOG=y
# CONFIG_GRKERNSEC_CHROOT_EXECLOG is not set
# CONFIG_GRKERNSEC_AUDIT_PTRACE is not set
# CONFIG_GRKERNSEC_AUDIT_CHDIR is not set
# CONFIG_GRKERNSEC_AUDIT_MOUNT is not set
CONFIG_GRKERNSEC_SIGNAL=y
# CONFIG_GRKERNSEC_FORKFAIL is not set
CONFIG_GRKERNSEC_TIME=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_RWXMAP_LOG=y
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_HARDEN_PTRACE=y
CONFIG_GRKERNSEC_PTRACE_READEXEC=y
CONFIG_GRKERNSEC_SETXID=y
CONFIG_GRKERNSEC_HARDEN_IPC=y
# CONFIG_GRKERNSEC_TPE is not set
CONFIG_GRKERNSEC_BLACKHOLE=y
CONFIG_GRKERNSEC_NO_SIMULT_CONNECT=y
# CONFIG_GRKERNSEC_SOCKET is not set
CONFIG_GRKERNSEC_DENYUSB=y
# CONFIG_GRKERNSEC_DENYUSB_FORCE is not set
CONFIG_GRKERNSEC_SYSCTL=y
# CONFIG_GRKERNSEC_SYSCTL_DISTRO is not set
CONFIG_GRKERNSEC_SYSCTL_ON=y
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=6

in the mean time i've installed and booted gentoo-sources and now lxc-create works fine as user.
however after setting /sys/fs/cgroup/memory/memory.use_hierarchy, /sys/fs/cgroup/cpuset/cgroup.clone_children and changing permissions on /sys/fs/cgroup/*, i'm stuck again trying to start the container:

Code: Select all

mybp@rock ~ $ lxc-start -n unpriv -f ~/.config/lxc/mybp.conf -l DEBUG -o mybp.log
newuidmap: write to uid_map failed: Invalid argument
lxc-start: start.c: lxc_spawn: 955 failed to set up id mapping
lxc-start: start.c: __lxc_start: 1080 failed to spawn 'unpriv'
lxc-start: lxc_start.c: main: 342 The container failed to start.
lxc-start: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options.

follows the log:

Code: Select all

      lxc-start 1455916894.089 INFO     lxc_utils - utils.c:get_rundir:409 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 1455916894.089 INFO     lxc_confile - confile.c:config_idmap:1325 - read uid map: type u nsid 0 hostid 100000 range 65536
      lxc-start 1455916894.089 INFO     lxc_confile - confile.c:config_idmap:1325 - read uid map: type g nsid 0 hostid 100000 range 65536
      lxc-start 1455916894.089 WARN     lxc_log - log.c:lxc_log_init:316 - lxc_log_init called with log already initialized
      lxc-start 1455916894.089 INFO     lxc_confile - confile.c:config_idmap:1325 - read uid map: type u nsid 0 hostid 100000 range 65536
      lxc-start 1455916894.089 INFO     lxc_confile - confile.c:config_idmap:1325 - read uid map: type g nsid 0 hostid 100000 range 65536
      lxc-start 1455916894.089 INFO     lxc_confile - confile.c:config_idmap:1325 - read uid map: type u nsid 0 hostid 100000 range 65536
      lxc-start 1455916894.089 INFO     lxc_confile - confile.c:config_idmap:1325 - read uid map: type g nsid 0 hostid 100000 range 65536
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpuset unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpu unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup cpuacct unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup blkio unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup memory unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup devices unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup freezer unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup net_cls unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup perf_event unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup net_prio unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 WARN     lxc_cgfs - cgfs.c:lxc_cgroup_get_container_info:1100 - Not attaching to cgroup hugetlb unknown to /var/lxc/mybp/.local/share/lxc unpriv
      lxc-start 1455916894.090 INFO     lxc_utils - utils.c:get_rundir:409 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 1455916894.090 DEBUG    lxc_conf - conf.c:lxc_create_tty:3665 - allocated pty '/dev/pts/2' (5/6)
      lxc-start 1455916894.090 DEBUG    lxc_conf - conf.c:lxc_create_tty:3665 - allocated pty '/dev/pts/3' (7/8)
      lxc-start 1455916894.090 DEBUG    lxc_conf - conf.c:lxc_create_tty:3665 - allocated pty '/dev/pts/4' (9/10)
      lxc-start 1455916894.090 DEBUG    lxc_conf - conf.c:lxc_create_tty:3665 - allocated pty '/dev/pts/5' (11/12)
      lxc-start 1455916894.090 INFO     lxc_conf - conf.c:lxc_create_tty:3676 - tty's configured
      lxc-start 1455916894.090 DEBUG    lxc_start - start.c:setup_signal_fd:247 - sigchild handler set
      lxc-start 1455916894.090 DEBUG    lxc_console - console.c:lxc_console_peer_default:500 - opening /dev/tty for console peer
      lxc-start 1455916894.090 INFO     lxc_caps - caps.c:lxc_caps_up:101 - Last supported cap was 36
      lxc-start 1455916894.090 DEBUG    lxc_console - console.c:lxc_console_peer_default:506 - using '/dev/tty' as console
      lxc-start 1455916894.090 DEBUG    lxc_console - console.c:lxc_console_sigwinch_init:179 - 332 got SIGWINCH fd 17
      lxc-start 1455916894.090 DEBUG    lxc_console - console.c:lxc_console_winsz:88 - set winsz dstfd:14 cols:172 rows:45
      lxc-start 1455916894.487 INFO     lxc_start - start.c:lxc_init:443 - 'unpriv' is initialized
      lxc-start 1455916894.488 DEBUG    lxc_start - start.c:__lxc_start:1058 - Not dropping cap_sys_boot or watching utmp
      lxc-start 1455916894.488 INFO     lxc_start - start.c:lxc_spawn:802 - Cloning a new user namespace
      lxc-start 1455916894.488 INFO     lxc_cgroup - cgroup.c:cgroup_init:62 - cgroup driver cgroupfs initing for unpriv
      lxc-start 1455916894.492 ERROR    lxc_start - start.c:lxc_spawn:955 - failed to set up id mapping
      lxc-start 1455916894.492 INFO     lxc_utils - utils.c:get_rundir:409 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 1455916894.597 ERROR    lxc_start - start.c:__lxc_start:1080 - failed to spawn 'unpriv'
      lxc-start 1455916894.597 INFO     lxc_utils - utils.c:get_rundir:409 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 1455916894.597 INFO     lxc_utils - utils.c:get_rundir:409 - XDG_RUNTIME_DIR isn't set in the environment.
      lxc-start 1455916894.598 ERROR    lxc_start_ui - lxc_start.c:main:342 - The container failed to start.
      lxc-start 1455916894.598 ERROR    lxc_start_ui - lxc_start.c:main:346 - Additional information can be obtained by setting the --logfile and --logpriority options.

skunk · Post by **skunk** » Mon Feb 22, 2016 5:23 pm

ok, got confused by the guide, i've left a note on the discussion tab...

mimosinnet · Post by **mimosinnet** » Tue Feb 23, 2016 4:08 pm

Skunk,

I have been using LXC for a while following a previous version of the wiki, and the Unprivileged Cointainers looks quite promising. Thanks a lot for the comments.

Cheers!

skunk · Post by **skunk** » Wed Feb 24, 2016 12:42 pm

you're welcome, however i'm still wondering if this is enough for a true unprivileged container and why i get "newuidmap: write to uid_map failed: Invalid argument" if i set subuids/subgids to 100000-165536 regardless having SUB_U/GID_MIN and SUB_U/GID_MAX setted with 100000 and 600100000 into /etc/login.defs...