[SOLVED] mullvad-daemon: missing cgroup net_cls

[sirlark]

I'm trying to get mullvad-daemon running on my laptop. From the output below, I figured that I was missing some kernel stuff, but /sys/fs/cgroups/net_cls exists and is populated. Either way,
I noticed a new kernel had become available via an emerge --sync. So I ran emerge -uNDav @world and then went and fiddled with the kernel config, then built the kernel with some additional stuff (adding some cgroups and some tunnelling stuff, dee diff below). The problem is that the new kernel won't boot. I use secure boot and it doesn't get beyond the "EFI stub ..." text at very early boot. I can' see anything obvious in the kernel changes that would prevent boot, unless cgroups would do it. But I did notice the new kernel is being compiled with gcc-15, whereas my old (booting) kernel was compiled with gcc-14

Have I done anything obviously stupid here?

mullvad-daemon output

Code: Select all

[mullvad_daemon::version][INFO] Starting mullvad-daemon - 2025.6-beta2 2025-05-07
[mullvad_daemon][INFO] Logging to /var/log/mullvad-vpn
[mullvad_daemon::management_interface][INFO] Management interface listening on /var/run/mullvad-vpn
[mullvad_daemon::settings][INFO] Loading settings from /etc/mullvad-vpn/settings.json
[mullvad_relay_selector::relay_selector::parsed_relays][ERROR] Failed to load bundled relays: Failed to open relay cache file
[mullvad_relay_selector::relay_selector][ERROR] Error: Unable to load cached and bundled relays
Caused by: Failed to open relay cache file
Caused by: No such file or directory (os error 2)
[mullvad_relay_selector::relay_selector][INFO] Initialized with 0 cached relays from 1970-01-01 01:00:00.000
[mullvad_daemon::account_history][INFO] Opening account history file in /etc/mullvad-vpn/account-history.json
[talpid_core::firewall][INFO] Resetting firewall policy
[mullvad_daemon::api][INFO] Initial offline state - online
[mullvad_daemon::version_check][WARN] Error: Unable to load cached version info
Caused by: Failed to open app version cache file for reading
Caused by: No such file or directory (os error 2)
[talpid_core::dns][INFO] Resetting DNS
[mullvad_daemon][ERROR] Error: Unable to initialize daemon
Caused by: Unable to initialize split tunneling
Caused by: Unable to initialize net_cls cgroup instance
Caused by: EINVAL: Invalid argument

kernel config diff

Code: Select all

--- ../linux-6.14.4-gentoo/.config	2025-04-26 20:05:47.941569783 +0100
+++ .config	2025-05-29 17:07:00.515382828 +0100
@@ -1,18 +1,18 @@
 #
 # Automatically generated file; DO NOT EDIT.
-# Linux/x86 6.14.2-gentoo Kernel Configuration
+# Linux/x86 6.14.7-gentoo Kernel Configuration
 #
-CONFIG_CC_VERSION_TEXT="gcc (Gentoo 14.2.1_p20250301 p8) 14.2.1 20250301"
+CONFIG_CC_VERSION_TEXT="gcc (Gentoo 15.1.0 p55) 15.1.0"
 CONFIG_CC_IS_GCC=y
-CONFIG_GCC_VERSION=140201
+CONFIG_GCC_VERSION=150100
 CONFIG_CLANG_VERSION=0
 CONFIG_AS_IS_GNU=y
 CONFIG_AS_VERSION=24400
 CONFIG_LD_IS_BFD=y
 CONFIG_LD_VERSION=24400
 CONFIG_LLD_VERSION=0
-CONFIG_RUSTC_VERSION=108501
-CONFIG_RUSTC_LLVM_VERSION=190107
+CONFIG_RUSTC_VERSION=108700
+CONFIG_RUSTC_LLVM_VERSION=200101
 CONFIG_CC_CAN_LINK=y
 CONFIG_CC_CAN_LINK_STATIC=y
 CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
@@ -20,9 +20,10 @@
 CONFIG_TOOLS_SUPPORT_RELR=y
 CONFIG_CC_HAS_ASM_INLINE=y
 CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
+CONFIG_CC_HAS_COUNTED_BY=y
 CONFIG_LD_CAN_USE_KEEP_IN_OVERLAY=y
 CONFIG_RUSTC_HAS_COERCE_POINTEE=y
-CONFIG_PAHOLE_VERSION=129
+CONFIG_PAHOLE_VERSION=130
 CONFIG_IRQ_WORK=y
 CONFIG_BUILDTIME_TABLE_SORT=y
 CONFIG_THREAD_INFO_IN_TASK=y
@@ -189,19 +190,27 @@
 CONFIG_GCC_NO_STRINGOP_OVERFLOW=y
 CONFIG_CC_NO_STRINGOP_OVERFLOW=y
 CONFIG_ARCH_SUPPORTS_INT128=y
+CONFIG_SLAB_OBJ_EXT=y
 CONFIG_CGROUPS=y
+CONFIG_PAGE_COUNTER=y
 # CONFIG_CGROUP_FAVOR_DYNMODS is not set
-# CONFIG_MEMCG is not set
-# CONFIG_BLK_CGROUP is not set
-# CONFIG_CGROUP_SCHED is not set
+CONFIG_MEMCG=y
+# CONFIG_MEMCG_V1 is not set
+CONFIG_BLK_CGROUP=y
+CONFIG_CGROUP_WRITEBACK=y
+CONFIG_CGROUP_SCHED=y
+CONFIG_GROUP_SCHED_WEIGHT=y
+CONFIG_FAIR_GROUP_SCHED=y
+CONFIG_CFS_BANDWIDTH=y
+# CONFIG_RT_GROUP_SCHED is not set
 CONFIG_SCHED_MM_CID=y
-# CONFIG_CGROUP_PIDS is not set
-# CONFIG_CGROUP_RDMA is not set
-# CONFIG_CGROUP_DMEM is not set
+CONFIG_CGROUP_PIDS=y
+CONFIG_CGROUP_RDMA=y
+CONFIG_CGROUP_DMEM=y
 # CONFIG_CGROUP_FREEZER is not set
 # CONFIG_CPUSETS is not set
 # CONFIG_CGROUP_DEVICE is not set
-# CONFIG_CGROUP_CPUACCT is not set
+CONFIG_CGROUP_CPUACCT=y
 # CONFIG_CGROUP_PERF is not set
 CONFIG_CGROUP_BPF=y
 # CONFIG_CGROUP_MISC is not set
@@ -854,8 +863,12 @@
 CONFIG_BLK_DEV_INTEGRITY=y
 CONFIG_BLK_DEV_WRITE_MOUNTED=y
 # CONFIG_BLK_DEV_ZONED is not set
+# CONFIG_BLK_DEV_THROTTLING is not set
 CONFIG_BLK_WBT=y
 CONFIG_BLK_WBT_MQ=y
+# CONFIG_BLK_CGROUP_IOLATENCY is not set
+# CONFIG_BLK_CGROUP_IOCOST is not set
+# CONFIG_BLK_CGROUP_IOPRIO is not set
 # CONFIG_BLK_SED_OPAL is not set
 # CONFIG_BLK_INLINE_ENCRYPTION is not set
 
@@ -1061,6 +1074,8 @@
 CONFIG_NET_INGRESS=y
 CONFIG_NET_EGRESS=y
 CONFIG_NET_XGRESS=y
+CONFIG_SKB_DECRYPTED=y
+CONFIG_SKB_EXTENSIONS=y
 CONFIG_NET_DEVMEM=y
 
 #
@@ -1071,9 +1086,17 @@
 CONFIG_UNIX=y
 CONFIG_AF_UNIX_OOB=y
 CONFIG_UNIX_DIAG=y
-# CONFIG_TLS is not set
+CONFIG_TLS=y
+CONFIG_TLS_DEVICE=y
+# CONFIG_TLS_TOE is not set
+CONFIG_XFRM=y
 # CONFIG_XFRM_USER is not set
+# CONFIG_XFRM_INTERFACE is not set
+# CONFIG_XFRM_SUB_POLICY is not set
+# CONFIG_XFRM_MIGRATE is not set
+# CONFIG_XFRM_STATISTICS is not set
 # CONFIG_NET_KEY is not set
+# CONFIG_XFRM_IPTFS is not set
 # CONFIG_XDP_SOCKETS is not set
 CONFIG_NET_HANDSHAKE=y
 CONFIG_INET=y
@@ -1084,17 +1107,19 @@
 # CONFIG_IP_ROUTE_MULTIPATH is not set
 # CONFIG_IP_ROUTE_VERBOSE is not set
 # CONFIG_IP_PNP is not set
-# CONFIG_NET_IPIP is not set
+CONFIG_NET_IPIP=y
 # CONFIG_NET_IPGRE_DEMUX is not set
 CONFIG_NET_IP_TUNNEL=y
 # CONFIG_SYN_COOKIES is not set
-# CONFIG_NET_IPVTI is not set
+CONFIG_NET_IPVTI=y
 CONFIG_NET_UDP_TUNNEL=y
 # CONFIG_NET_FOU is not set
+# CONFIG_NET_FOU_IP_TUNNELS is not set
 # CONFIG_INET_AH is not set
 # CONFIG_INET_ESP is not set
 # CONFIG_INET_IPCOMP is not set
 CONFIG_INET_TABLE_PERTURB_ORDER=16
+CONFIG_INET_TUNNEL=y
 CONFIG_INET_DIAG=m
 CONFIG_INET_TCP_DIAG=m
 CONFIG_INET_UDP_DIAG=m
@@ -1194,6 +1219,7 @@
 CONFIG_NFT_HASH=m
 CONFIG_NFT_FIB=m
 # CONFIG_NFT_FIB_INET is not set
+# CONFIG_NFT_XFRM is not set
 CONFIG_NFT_SOCKET=m
 # CONFIG_NFT_OSF is not set
 CONFIG_NFT_TPROXY=m
@@ -1219,7 +1245,7 @@
 #
 # CONFIG_NETFILTER_XT_TARGET_CHECKSUM is not set
 # CONFIG_NETFILTER_XT_TARGET_CLASSIFY is not set
-# CONFIG_NETFILTER_XT_TARGET_CONNMARK is not set
+CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
 # CONFIG_NETFILTER_XT_TARGET_CT is not set
 # CONFIG_NETFILTER_XT_TARGET_DSCP is not set
 # CONFIG_NETFILTER_XT_TARGET_HL is not set
@@ -1275,6 +1301,7 @@
 # CONFIG_NETFILTER_XT_MATCH_NFACCT is not set
 # CONFIG_NETFILTER_XT_MATCH_OSF is not set
 # CONFIG_NETFILTER_XT_MATCH_OWNER is not set
+# CONFIG_NETFILTER_XT_MATCH_POLICY is not set
 # CONFIG_NETFILTER_XT_MATCH_PKTTYPE is not set
 # CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
 # CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
@@ -1410,8 +1437,8 @@
 CONFIG_RFS_ACCEL=y
 CONFIG_SOCK_RX_QUEUE_MAPPING=y
 CONFIG_XPS=y
-# CONFIG_CGROUP_NET_PRIO is not set
-# CONFIG_CGROUP_NET_CLASSID is not set
+CONFIG_CGROUP_NET_PRIO=y
+CONFIG_CGROUP_NET_CLASSID=y
 CONFIG_NET_RX_BUSY_POLL=y
 CONFIG_BQL=y
 CONFIG_BPF_STREAM_PARSER=y
@@ -1514,6 +1541,7 @@
 # CONFIG_LWTUNNEL is not set
 CONFIG_DST_CACHE=y
 CONFIG_GRO_CELLS=y
+CONFIG_SOCK_VALIDATE_XMIT=y
 CONFIG_NET_SELFTESTS=y
 CONFIG_NET_SOCK_MSG=y
 CONFIG_PAGE_POOL=y
@@ -4611,6 +4639,7 @@
 # CONFIG_HID_THRUSTMASTER is not set
 # CONFIG_HID_UDRAW_PS3 is not set
 # CONFIG_HID_U2FZERO is not set
+# CONFIG_HID_UNIVERSAL_PIDFF is not set
 # CONFIG_HID_WACOM is not set
 # CONFIG_HID_WIIMOTE is not set
 # CONFIG_HID_WINWING is not set
@@ -6378,6 +6407,7 @@
 CONFIG_F2FS_FS_LZ4HC=y
 CONFIG_F2FS_FS_ZSTD=y
 # CONFIG_F2FS_IOSTAT is not set
+# CONFIG_F2FS_UNFAIR_RWSEM is not set
 # CONFIG_BCACHEFS_FS is not set
 # CONFIG_FS_DAX is not set
 CONFIG_FS_POSIX_ACL=y
@@ -6949,16 +6979,19 @@
 CONFIG_CRYPTO_LIB_GF128MUL=y
 CONFIG_CRYPTO_ARCH_HAVE_LIB_BLAKE2S=y
 CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC=y
-CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
+CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=y
 CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
+CONFIG_CRYPTO_LIB_CHACHA_INTERNAL=m
 CONFIG_CRYPTO_LIB_CHACHA=m
-CONFIG_CRYPTO_ARCH_HAVE_LIB_CURVE25519=m
+CONFIG_CRYPTO_ARCH_HAVE_LIB_CURVE25519=y
 CONFIG_CRYPTO_LIB_CURVE25519_GENERIC=m
+CONFIG_CRYPTO_LIB_CURVE25519_INTERNAL=m
 CONFIG_CRYPTO_LIB_CURVE25519=m
 CONFIG_CRYPTO_LIB_DES=y
 CONFIG_CRYPTO_LIB_POLY1305_RSIZE=11
-CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305=m
+CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305=y
 CONFIG_CRYPTO_LIB_POLY1305_GENERIC=m
+CONFIG_CRYPTO_LIB_POLY1305_INTERNAL=m
 CONFIG_CRYPTO_LIB_POLY1305=m
 CONFIG_CRYPTO_LIB_CHACHA20POLY1305=m
 CONFIG_CRYPTO_LIB_SHA1=y

[pietinger]

It is not clear from your report whether the mullvad daemon has ever been running.

If so, please boot with this old kernel and give us the complete [1] configuration of this old kernel. Also explain how [2] you did the configuration of the new kernel. The fact that you cannot boot the new kernel probably has another cause than “cgroups”. Has it been signed correctly?

If not: We will need the complete output of “dmesg” in any case (as soon as the kernel has been signed correctly and then boots), as well as the complete kernel configuration.

[1] Use “wgetpaste” for this; see https://wiki.gentoo.org/wiki/User:Pieti ... nformation
[2] https://wiki.gentoo.org/wiki/User:Pieti ... it_.config
+ https://wiki.gentoo.org/wiki/User:Pieti ... fig.22_.3F

[sirlark]

mullvad-daemon never ran, even on previous kernels that booted.

Let me outline my exact steps

1. I have a working kernel booted (Linux 6.14.4-gentoo)
2. I copy it's .config into /usr/src/linux (which points to /usr/src/linux-6.14.7-gentoo) That config can be seen here
3. I run `make oldconfig` (there's only one new config item for a sensor I don't use, so I answer NO)
4. I run `make menuconfig` and make the modifications outlined in the diff above
5. I compile the kernel (make -j 17 && make -j 17 modules && make modules_install)
6. I copy the bzimage to /boot, and sign it to produce /boot/EFI/Gentoo/bzImage-gentoo-6.14.7.efi.signed)
7. I run efibootmgr to add a boot entry
8. Reboot

I have a script that I use to sign newly compiled kernels, which is the same script I used for the kernel that boots successfully, so I don't think it's a signing issue, but I will confirm this by skipping step 4 above, to see if I can boot into a signed 6.14.7 kernel without anything but `make oldconfig` run

[pietinger]

[...] 5. I compile the kernel (make -j 17 && make -j 17 modules && make modules_install)

Usually make -j X & make modules_install is sufficient (the make creates/handles also all modules).

6. I copy the bzimage to /boot, and sign it to produce /boot/EFI/Gentoo/bzImage-gentoo-6.14.7.efi.signed)

I'm not sure if this works, because as far as I know a bootable efi-executable must end with *.efi - without *.efi.XXX

I do almost the same thing ... except that I leave the copying to sbsign ... ->
https://wiki.gentoo.org/wiki/User:Pieti ... SecureBoot
(I dont make a new UEFI entry every time; I use only two again and again)

mullvad-daemon never ran, even on previous kernels that booted.

I guess you miss some options. Does it work if you boot our distribution kernel (and chroot into your system) ? If yes, then do a "lsmod" and check all loaded modules.

I guess it all starts with enabling # CONFIG_NET_SCHED is not set


BTW: It looks like you have taken the config from our dist-kernel and modified it (it has hundreds of modules you will never need; maybe time to make an own kernel configuration; see my articles) ... and also CONFIG_GENTOO_KERNEL_SELF_PROTECTION=y gives you nothing (see my KSPP article); e.g you miss # CONFIG_KFENCE is not set (and many more).

[sirlark]

Actually, I did a kernel config from scratch. I've just got all those extra modules because I'm still trying to identify the specific drivers for bits and pieces of my own hardware. This laptop has been a REAL PAIN to get working. I did have CONFIG_KFENCE set at some point, and again got a non-booting kernel, so there's some left over kruft for sure.

The currently working kernel boots fine from .efi.signed

[pietinger]

[...] This laptop has been a REAL PAIN to get working. [...]

I know the pain ... ... because I had learned that you dont see everything in "lsmod" ... because some drivers are configured built-in (the dist.kernel; Ubuntu kernel is better -> more modules as <M> configured but also not all) ... so, yes it is hard to identify everything you need ... only one example (of some): https://wiki.gentoo.org/wiki/User:Pieti ... nstantiate

[...] The currently working kernel boots fine from .efi.signed

Thanks a lot for this ... good to know.

[pietinger]

P.S.: I saw that this was disabled:

Code: Select all

[*]   CPU core priorities scheduler support

... maybe because you want disable Intel P-State ... BUT ... dont do it ... if you want disable Intel P-State you must disable Intel support ... for this you must enable CONFIG_EXPERT (see also in my article).

[sirlark]

Okay! So a freshly compiled 6.14.7 kernel with no changes from the 6.14.4 config other than what make oldconfig did boots fine. It's definitely an issue with what I did messing around with config. Now back to the mullvad thing

[pietinger]

Maybe go immediately to 6.14.9-r1 (because you have an AMD machine) before doing some configurations ... ?

[sirlark]

Okay, something funky definitely happened to my previous compile of 6.14.7; `modprobe cls_cgroup` failed with "Exec Format Error". After I did a `make clean && make mrproper` along with `rm -rf /lib/modules/6.14.7-gentoo`. After a reboot into the newly compiled kernel, the modprobe works, and mullvad-daemon starts up. I can also use mullvad-vpm client successfully.

Thanks for the help

[pietinger]

Thanks for the help

Oh, I don't think I was that much help ... and you are of course very Welcome.

[sirlark]

Actually, you're mention of CONFIG_NET_SCHED tipped me off to the minimal change required to get mullvad working. It allowed me to rule out all of the other confounding variables.

[pietinger]

Actually, you're mention of CONFIG_NET_SCHED tipped me off to the minimal change required to get mullvad working. It allowed me to rule out all of the other confounding variables.

Yes, that was my evil plan

(Happy to hear that)