How to bond network interfaces?

[orion777]

I'm newbee, sorry about that. I'm very interested in two interfaces bonding. The primary consideration to increase availability of the connection, however, I would like to try network balancing to.
The key problem, that I'm found a couple of how-to use ifenslave, but most of them are 10 years old and reports DIFFERENT configuration! Some of them require fully manual config of the network (what, seems, is ignored in my system), other require kernel config; miimon and bonding type specification is aso different..

• for example:
  https://syzygytech.com/?p=106
  viewtopic-t-198607-start-0.html
  https://habrahabr.ru/post/58218/
  https://wiki.archlinux.org/index.php/Ne ... 0.BE.D0.B2
  etc

That why I'm asking about bond config here; to make it clear and to exclude possibility to make my device unbootable (raspberry pi3b)

What I have:
- rpi-4.10 kernel
- running dhcpcd
- (optionally NetworkManager and/or dhclient CAN be runned, but these are not in autostart for now)
- ls /sys/class/net reports my interfaces correctly; also ifconfig reports all my network interfaces
- config in /etc/conf.d/net is ignored (at least I was trying to specify static IP configuration for the eth1, but eth1 still gets its ip by means of dhcpcd)

[khayyam]

The key problem, that I'm found a couple of how-to use ifenslave, but most of them are 10 years old and reports DIFFERENT configuration! Some of them require fully manual config of the network (what, seems, is ignored in my system), other require kernel config; miimon and bonding type specification is aso different.

orion777 ... a read of the section on bonding in /usr/share/doc/netifrc-*/net.example.bz2 and /usr/src/linux-*/Documentation/networking/bonding.txt should be all that's needed, and note that 'ifenslave' is for "2.4 kernels, or kernels without sysfs". There is also the bonding section on the wiki with a concete example.

HTH & best ... khay

[orion777]

The key problem, that I'm found a couple of how-to use ifenslave, but most of them are 10 years old and reports DIFFERENT configuration! Some of them require fully manual config of the network (what, seems, is ignored in my system), other require kernel config; miimon and bonding type specification is aso different.

orion777 ... a read of the section on bonding in /usr/share/doc/netifrc-*/net.example.bz2 and /usr/src/linux-*/Documentation/networking/bonding.txt should be all that's needed, and note that 'ifenslave' is for "2.4 kernels, or kernels without sysfs". There is also the bonding section on the wiki with a concete example.

HTH & best ... khay

Yes, I know this page to. However, I can't understand, ifenslave and bonding (described in wiki.gentoo.org/wiki/Handbook:Parts/Networking/Modular/en#Bonding ) it is fully different stuff?
ifenslave is not possible in kernel 4.10? (I can't run 2.4)

You specify the wiki page, that use /etc/conf.d/net to configure bonded interfaces, however, my /etc/conf.d/net seems ignored, as static ip configured there is ignored..

[khayyam]

[...] I can't understand, ifenslave and bonding (described in wiki.gentoo.org/wiki/Handbook:Parts/Networking/Modular/en#Bonding ) it is fully different stuff? ifenslave is not possible in kernel 4.10? (I can't run 2.4)

orion777 ... forget about ifenslave, this is depreciated, iproute2 was written to support all such kernel networking features (vlan, bonding, etc).

You specify the wiki page, that use /etc/conf.d/net to configure bonded interfaces, however, my /etc/conf.d/net seems ignored, as static ip configured there is ignored..

You'll have to show ...

best ... khay

[orion777]

I have no running NetworkManager, only dhcpcd is in autostart. eth1 is attached via usb.
Here it is:

Code: Select all

pi64 ~ # uname -a
Linux pi64 4.10.17-v8+ #3 SMP PREEMPT Fri Dec 15 11:38:53 GMT 2017 aarch64 GNU/Linux
pi64 ~ # ps -A | grep NetworkManager
pi64 ~ # ps -A | grep dhclient
pi64 ~ # ps -A | grep dhcpcd
 1718 ?        00:00:00 dhcpcd
pi64 ~ # ifconfig eth1
eth1: error fetching interface information: Device not found
pi64 ~ # cat /etc/conf.d/net
config_eth1="192.168.8.15/24"
routes_eth1="default via 192.168.8.1"
dns_servers_eth1="192.168.8.1"
pi64 ~ # echo "inserting USB eth1"
inserting USB eth1
pi64 ~ # ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.8.10  netmask 255.255.255.0  broadcast 192.168.8.255
        inet6 fe80::5d49:fee0:c509:bb11  prefixlen 64  scopeid 0x20<link>
        ether 0c:5b:8f:27:9a:64  txqueuelen 1000  (Ethernet)
        RX packets 27  bytes 9575 (9.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1626 (1.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

As You can see, eth1 gets its ip by means of dhcp, while config_eth1="192.168.8.15/24 was ignored.

And here is same situation, when dhcpcd also has been stopped (after reboot):

Code: Select all

pi64 ~ # ps -A | grep dhcpcd
 1715 ?        00:00:00 dhcpcd
pi64 ~ # kill -9 1715
pi64 ~ # ps -A | grep dhcpcd
pi64 ~ # ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.118  netmask 255.255.255.0  broadcast 192.168.1.255
        ether b8:27:eb:28:f1:db  txqueuelen 1000  (Ethernet)
        RX packets 148  bytes 13032 (12.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 94  bytes 15241 (14.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 24  bytes 1728 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24  bytes 1728 (1.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether b8:27:eb:7d:a4:8e  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi64 ~ # echo "inserting eth1"
inserting eth1
pi64 ~ # ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.118  netmask 255.255.255.0  broadcast 192.168.1.255
        ether b8:27:eb:28:f1:db  txqueuelen 1000  (Ethernet)
        RX packets 221  bytes 18036 (17.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 133  bytes 21495 (20.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 24  bytes 1728 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24  bytes 1728 (1.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether b8:27:eb:7d:a4:8e  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi64 ~ # ifconfig eth1 up

pi64 ~ # ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.118  netmask 255.255.255.0  broadcast 192.168.1.255
        ether b8:27:eb:28:f1:db  txqueuelen 1000  (Ethernet)
        RX packets 285  bytes 22282 (21.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 172  bytes 28101 (27.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e5b:8fff:fe27:9a64  prefixlen 64  scopeid 0x20<link>
        ether 0c:5b:8f:27:9a:64  txqueuelen 1000  (Ethernet)
        RX packets 11  bytes 5237 (5.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9  bytes 746 (746.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 24  bytes 1728 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24  bytes 1728 (1.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether b8:27:eb:7d:a4:8e  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Please note that without running dhcpcd eth1 was not UP automatically. Also ip was not assigned (/etc/conf.d/net was ignored again)

[khayyam]

orion777 ...

that all seems consistent to me (if a little difficult to figure out what you're doing). If you want dhcpcd to ignore the interface then add '--denyinterfaces <pattern>' as a parameter, or config, for dhcpcd, eg:

Code: Select all

denyinterfaces eth1

The problem you have in the above is that as dhcpcd is started it is grabing the interface(s). I really don't see why you have dhcpcd in the runlevel at all, when you are starting net.eth1 (if in fact you are).

HTH & best ... khay

[orion777]

I have to use two usb cellular 3G/4G dongles huawei 3372h simultaneously. The problem is that all 3372h have same mac-address and these cannot be changed (this problem is well known and was discussed in cellular modem forum), also you can take a look at udevadm when both dongles are attached, here in post #3 viewtopic-t-1072550.html
By default, NetworkManager starts dhclient for the first inserted dongle, while second inserted was ignored and ip was not assigned by dhclient (probably due of similar mac-addresses). Than I was disabling NetworkManager (hence there are also no child process dhclient), than dhcpcd was added into autorun and now both dongles gets their ip by means of dhcpcd (by the cost that DNS server is not working and all requests should be done thru ip, hence emerge,etc cant recognize servers and are not working).

WHAT I WANT:
I want to connect two 3G/4G dongles and mate them together to get reliable solution (by means of parallel redundant network) for moving equipment (remotely controlled car-robot with camera, that why I'm using raspberry pi 3b with sakakis' rpi64 image https://github.com/sakaki-/gentoo-on-rpi3-64bit ). Dongles have different ip and gateway addresses, probably these can be configured manually to exclude dhcp problem, but these MUST be assigned based on eth number, NOT mac-address (as ethers are same). Dongles eth numbers are assigned by USB port number, hence are same all the time

Code: Select all

pi64 /etc/udev/rules.d # cat 70-etc3.rules
KERNELS=="1-1.4:1.0", NAME="eth3"

Ok ,maybe I'm completely wrong.. Who is responsible to take config from /etc/conf.d/net ? Kernel, NetworkManager?

[szatox]

I want to connect two 3G/4G dongles and mate them together to get reliable solution (by means of parallel redundant network) for moving equipment

IMO it's a no-no, or at least it's not a as simple as stuffing 2 dongles next to each other.
You won't be able to move a single connection from one dongle to the other on the fly anyway. How long downtime is too long for you? Do you need fault tolerance, or can you afford simply reconnecting?
What kind of issues do you want to guard from?
Also what interface those modems expose to the system? A glance at the other topic suggests it's ethernet, not the typical ppp over serial. How much control you have over this device? Is it essentially an independent router you have to configure via it's fake website?
How you get an IP for those interfaces? Is it configurable in the device, completely random, or maybe they always give you the same IP (on both interfaces)?

[orion777]

IMO it's a no-no, or at least it's not a as simple as stuffing 2 dongles next to each other.
You won't be able to move a single connection from one dongle to the other on the fly anyway. How long downtime is too long for you? Do you need fault tolerance, or can you afford simply reconnecting?

This solution should be installed on remotely controlled vehicle. I want to replace conventional line-of-sight limited radio control by means of gsm. This means that such gsm solution must be stable, without loss of connection. Dongles should be installed on remotely controlled vehicle only. Something that "parallel redundancy", when packet is send over two interfaces in parallel OR whep packet is send thru one interface, but in a case of ~1 sec delay it start to use second interface to send data.

What kind of issues do you want to guard from?

Data communication loss, probably due of to weak signal. Both dongles are using different cellular operators, so they have different coverage to.

Also what interface those modems expose to the system? A glance at the other topic suggests it's ethernet, not the typical ppp over serial. How much control you have over this device? Is it essentially an independent router you have to configure via it's fake website?

These modems can operate in hilink (sw starting from 22.xxxxxxx) or stick whet these are PPP (sw 21.xxxx...). I'm using hilink mode. In tis mode donle emulates its virtual 150 Mbps ethernet port and local ip address (192.168.x.x) is assigned by dongles' dhcp server. Also in this mode dongle runs its internal NAT server and all configuration can be done by accessing its' web server (default ip 192.168.8.1), ^at commands are not possible in hilink since pc-ui port is not accessible in this mode, hence it is not possible to send a command to restart, reconnect, connect to 3G instead 4G, etc..

How you get an IP for those interfaces? Is it configurable in the device, completely random, or maybe they always give you the same IP (on both interfaces)?

By means of dhcp servers, that are running on both dongles. By default both dongles have same ip, so these are reconfigured to use different self-ip (that becomes gateway ip 192.168.8.1 and 192.168.9.2 respectively) and different DHCP ranges.

[szatox]

Alright.... Well, I have bad feelings about this setup, but you could try to circumvent some limitations of public networks by means of an overlay network.
Say, you could try setting up 2 VPN connections (one over each GSM link) to a single server. and then connect tap devices with a bridge or a bond.
There was even a kernel driver for fault tolerant networks that would broadcast your packet via all interfaces and only pick the first copy on the receiving end, dropping the rest, so changing paths does not add any latency at all. Sorry, can't recall its name right now.

Using those 2 VPNs or tunnels or whatever separates your transport layer from your public IP and lets you create a single IP space spanning both those links. Since your connection (source IP, source port, destination IP, destination port and sequence number) are not affected by the underlying GSM connection anymore, so you can unplug GSM without immediately breaking your connection in the overlay, so you can either re-establish that tunnel or switch to the other tunnel with exactly the same set of IPs at its ends.

I've never tried this myself, but you can try e.g. GRE tunnels or openvpn in bridged mode.
Also, you will need some tricks to do the routing properly. You will probably connect from your vehicle to a single remote IP. It calls for source-based routing (can be done with multiple routing tables, linux kernel supports this). And then you have to open each connection from a manually selected local IP rather than using the default.

I still I don't really like your idea, there are a lot of things than can go wrong, but you may research the above approach if you feel adventurous.

[orion777]

Thanks for the idea.

But I'm still is confusing: Who is responsible to read /etc/conf.d/net ?? As specified here https://wiki.gentoo.org/wiki/Handbook:P ... e_handlers preliminary network configuration is done there, but if /etc/conf.d/net is empty, than it will use dhcp by default.

My /etc/conf.d/net was created manually, because by default net was not exist in my gentoo. Maybe here I was wrong?
Because static IP config is ignored for build-in ethernet card to (now NetworkManager is running)!

Code: Select all

pi64 ~ # cat /etc/conf.d/net
config_eth0="192.168.1.15/24"
pi64 ~ # ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.118  netmask 255.255.255.0  broadcast 192.168.1.255
        ether b8:27:eb:28:f1:db  txqueuelen 1000  (Ethernet)
        RX packets 144  bytes 12348 (12.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 92  bytes 14239 (13.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[Moderator edit: added [code] tags to preserve output layout. -Hu]

[szatox]

/etc/conf.d/net is read by netifrc.
You can invoke it by linking /etc/init.d/net.<your interface name> to /etc/init.d/net.lo and then calling the service by newly created name.
So, if your interface is called eth0, you do cd /etc/init.d; ln -s net.lo net.eth0; ./net.eth0 start
and then it will load variables defined in /etc/conf.d/net and apply those related to eth0.

As I think of it now, you may have to define dependencies by yourself, to ensure openrc starts network interfaces in the correct order.
Also, setups with redundant connections are prone to loops. If you find out your network misbehaves, remove one link and see if it gets better.

[orion777]

/etc/conf.d/net is read by netifrc.
You can invoke it by linking /etc/init.d/net.<your interface name> to /etc/init.d/net.lo and then calling the service by newly created name...

That was my mistake... When ln is created and added into autostart, NetworkManager can be removed from the autostart and interface will gets its ip as specified in /etc/conf.d/net, Thank You!!

Now I was trying to bond two network cards together, but something going wrong, as cards report NO network activity at all.. (by means of cards indicators; ifconfig; bond ip address also does not reply on ping)

Here is my configs:

Code: Select all

pi64 /etc/init.d # ls | grep net
net-online
net.bond0
net.eth1
net.eth3
net.eth4
net.lo
netmount

pi64 /etc/init.d # cat /etc/conf.d/net
config_eth3="null"
config_eth4="null"
slaves_bond0="eth3 eth4"
config_bond0="192.168.8.100/24"
mode_bond0="balance-alb"

pi64 /etc/init.d # rc-update | grep net
                local |      default nonetwork
           net.bond0 |      default
             netmount |      default

pi64 /etc/init.d # ifconfig bond0
bond0: flags=5123<UP,BROADCAST,MASTER,MULTICAST>  mtu 1500
        inet 192.168.8.100  netmask 255.255.255.0  broadcast 192.168.8.255
        ether 26:a1:2f:7b:18:86  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

"<M> Bonding driver support" in kernel .config is also implemented..

[szatox]

Code: Select all

slaves_bond0="eth3 eth4" 
config_bond0="192.168.8.100/24" 

Bonding 2 GSM (or actually any sort of customer connection to the public network) won't work. You have created a bond going to 2 different networks. Eth3 goes to one modem, eth4 to the other right? So... These modems don't know about each other and have no way to agree on their internal state.
That's why I mentioned setting up VPN or any other tunnel to a server on the other side of the internet, and then bonding the endpoints of your tunnels.
You need that additional layer to detach your traffic from any particular physical connection. A tunnel acts like a wire, it will carry any traffic you pump into it, even if it's invalid (any sort of balancing bond will send invalid traffic to each link), but the other machine will see data coming from both links, so you can create another bond there and it will put the shit together, so your data stream becomes complete and correct again.

[orion777]

Yep, seems that You are right..
I was trying to set the following configuration

eth3 -> 192.168.8.* - NAT (gateway 192.168.8.1) -> external ip
----------------------------------------------------------------------------------------> cellular provider server 1020.237.42 -> ... wan ...
eth4 -> 192.168.8.* - NAT (gateway 192.168.8.1) -> external ip

eth3 and eth4 was bonded with ip 192.168.8.100 default gateway 192.168.8.1, however /etc/init.d/net.bond8 start returns error on line 170, write error for both eth3 and eth4.

[orion777]

So, what next?
1) is it hand to setup two vpn from cmd (without x-servers) ?
2) maybe I have to try http://multipath-tcp.org/ ?
I'm noob in linux, unfortunately

[szatox]

That multipath tcp looks interesting, I think I've seen something like that running on an enterprizy server, though it was called by different name. Unfortunately I do not know nearly enough details to estimate its usefulness for your case. I've seen it running within a local network.
Anyway, in linux everything can be done from command line without Xserver.
You're trying to do something quite tricky there, so whichever option you chose, you will have to do your own research on this topic. It's certainly going to be a funny little project, but you can't really skip the research part. Good news is, it's gonna make you less noob

For GRE this one looks promissing
http://lartc.org/howto/lartc.tunnel.gre.html
In case of openvpn you'd have to write your own config file. It's not the most basic setup though, so my guess is gre would be easier.