Getting started with Incus

[o5gmmob8]

I would like to migrate my FreeBSD jails to Incus on Gentoo.

I would like to preserve my setup roughly:

Code: Select all

host
  workstation (graphical (X11 / wayland)
  router (2 Network Interfaces are assigned to it, WAN and LAN)

I am following this guide:
https://wiki.gentoo.org/wiki/Incus

If I understand correctly, wayland will be a bit trickier than X11:

Code: Select all

echo "gui:1000:1" | sudo tee -a /etc/subuid /etc/subgid

It is worth nothing that I have not assigned any video cards over to the workstation, but also, my system is a 3rd generation intel with just onboard video, so I don't have anything to allocate anyways.


The way I setup my FreeBSD host was a barebones install so that way the attack surface is as small as possible and more importantly, I have as few updates or patches to apply so I don't need to reboot. The host is not graphical and has no network interfaces assigned to it as it has assigned those to the router. The router is responsible for handling all network functionality and for the host and workstation to have network functionality, essentially a connection is made between the host and router and the workstation and router.

Additionally, the host in my case does not have a GUI whatsoever installed, yet I am still able to launch X11 via startx.

I have my inittab configured on FreeBSD to make the first 2 terminals available to the host, next 3 for the router, and next 3 for the workstation. I am presently using:

Code: Select all

"/usr/sbin/jexec workstation /usr/libexec/getty Pc"             xterm   onifexists secure       "/usr/libexec/getty Pc"             xterm   onifexists secure

I presume that I will use something similar to:

Code: Select all

"incus exec workstation /usr/bin/agetty"             xterm   onifexists secure       "/usr/bin/agetty"             xterm   onifexists secure

to open a getty for that container so I could log in directly to it.

I don't see many forum posts on incus and am wondering how much it is used as well. Is what I want to do reasonable?

[o5gmmob8]

I think I have the containers sorted out and it seems easy enough to assign physical network interfaces to the container. It also seems to work fairly similarly to FreeBSD jails which makes my life easier.

I haven't crossed the bridge yet about running a graphical environment in incus, but for FreeBSD, I don't believe I did anything special at all. In both cases, my host system was a barebones install.

All the documentation I've seen thus far appear to be referencing running a graphical container from within a graphical host, but not on a barebones host. Perhaps it will be similar, no special configuration required.

[o5gmmob8]

I am configuring incus in my target gentoo installation prior to reboot as I would prefer if possible to install and configure it in a single go. That said, I am encountering an error while setting up the networking and from what I gathered, I need to have NAT supported loaded. For that, I have loaded nft_chain_nat and nf_nat.

Yet, the error remains:

Code: Select all

Error: Failed to create local member network "incusbr0" in project "default": Failed clearing firewall: Failed to list IPv4 rules (table filter)

Code: Select all

cat /tmp/incus | incus admin init --debug --preseed

Code: Select all

config: {}
networks:
- config:
    ipv4.address: auto
    ipv6.address: auto
  description: ""
  name: incusbr0
  type: ""
  project: default
storage_pools:
- config:
    source: z_500.4/incus
  description: ""
  name: default
  driver: zfs
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      network: incusbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null

[Hu]

The error references the filter table, not the nat table. Do you have netfilter support for the filter table? What is the output of zgrep IP_NF_ /proc/config.gz?

[o5gmmob8]

Ah, good call. This is the kernel from the livecd, I have not yet rebooted as I am trying to build my system entirely before rebooting into it.

Code: Select all

livecd ~ # zgrep IP_NF_ /proc/config.gz
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_SYNPROXY=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

EDIT:
I just ran:

Code: Select all

modprobe iptable_filter
modprobe br_netfilter

And then am trying to rerun the script setting up incus containers.

EDIT:
With that, I now get ipv6 table, so I need to load ipv6 ...

EDIT:

Code: Select all

modprobe ip6table_filter.ko

Awesome, that worked, so I will modify my script to load those modules prior to attempting to create a container.

EDIT:
I see this when trying to launch the container:

livecd /tmp # incus info --show-log router

Code: Select all

Name: router
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2025/04/20 20:20 EDT
Last Used: 2025/04/20 20:21 EDT

Log:

lxc router 20250421002145.250 ERROR    start - ../lxc-6.0.3/src/lxc/start.c:lxc_spawn:1738 - Operation not permitted - Failed to clone a new set of namespaces
lxc router 20250421002145.250 ERROR    start - ../lxc-6.0.3/src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "router"
lxc router 20250421002145.250 ERROR    lxccontainer - ../lxc-6.0.3/src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc 20250421002145.280 ERROR    af_unix - ../lxc-6.0.3/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20250421002145.280 ERROR    commands - ../lxc-6.0.3/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

I have gotten further with incus on an actual installation, but I want this to be fully setup during my system build / install process so I can just reboot into my complete install and have everything setup in a single go.

I looked here: https://discuss.linuxcontainers.org/t/s ... -a-pty/219

But, the error appears to be pertaining to the permissions on the Unix socket.

[o5gmmob8]

I am still getting the same error even when trying to setup incus in a chroot on a fully functional host. The only reason I'm testing it this way is that I can test my automated installer while using my computer as I would normally. In any case, the end result is the same:

Code: Select all

lxc router 20250502113032.535 ERROR    start - ../lxc-6.0.3/src/lxc/start.c:lxc_spawn:1738 - Operation not permitted - Failed to clone a new set of namespaces
lxc router 20250502113032.536 ERROR    start - ../lxc-6.0.3/src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "router"
lxc router 20250502113032.536 ERROR    lxccontainer - ../lxc-6.0.3/src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc 20250502113032.562 ERROR    af_unix - ../lxc-6.0.3/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20250502113032.562 ERROR    commands - ../lxc-6.0.3/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

Again, to summarize, I am attempting to run my automated installer to provision a new gentoo instance on a separate disk while running my main gentoo installation. The new instance is mounted at /mnt/gentoo and is a barebones install that will ideally be running 2 containers. 1 container would be for my workstation needs running a graphical environment, and the other, my networking needs, handling all networking and firewalling for my home network. My goal is to have this provisioned in a single shot, so I can pull the drive and then install it into the live system (I have multiple identical machines and use a backup machine as a build machine).

[pingtoo]

Again, to summarize, I am attempting to run my automated installer to provision a new gentoo instance on a separate disk while running my main gentoo installation. The new instance is mounted at /mnt/gentoo and is a barebones install that will ideally be running 2 containers. 1 container would be for my workstation needs running a graphical environment, and the other, my networking needs, handling all networking and firewalling for my home network. My goal is to have this provisioned in a single shot, so I can pull the drive and then install it into the live system (I have multiple identical machines and use a backup machine as a build machine).

So is it in your goal "incus" is must? or you just want to setup two containers?

[o5gmmob8]

I am modifying my setup / installation process now to reboot after installing everything else, then on reboot, it will continue with incus.

1. It would be ideal if I could do everything before as it means it is more hands off.
2. Right now, yes, I have just 2 containers, but I could foresee in the future adding another container for gaming possibly or isolating ollama.

[pingtoo]

I am modifying my setup / installation process now to reboot after installing everything else, then on reboot, it will continue with incus.

1. It would be ideal if I could do everything before as it means it is more hands off.
2. Right now, yes, I have just 2 containers, but I could foresee in the future adding another container for gaming possibly or isolating ollama.

Would it be easier just use Docker for container? You are less likely encounter the namespace issue.

But if your goal is about develop a "procedure" with specific tools in mind. then may be share your vision of the procedure would be much more helpful.

I am no user of incus, but from I learn from the website, it seems just a virtualization management tools. so if you want to develop the "procedure" with incus then you need to describe it in detail in your process how it got invoked. The namespace issue have something to do with possibly non-privileged UID invocation or missing configuration in host kernel.

[o5gmmob8]

Fair points - yes, I think I want to stick to incus over docker, but really, whatever works for those use cases above.

That said, this is what failed to run:

Code: Select all

incus launch images:gentoo/openrc router

Upon further inspection using the show logs cmd, that is where I saw the namespace issue.

I was talking high-level to see if my process itself was flawed. That said, I am rerunning my installer with the updated process (where I have to reboot into the new system) and then it will attempt to provision the containers. So, I will see how that works and what issues I run into.