MPTCP on Gentoo kernel 4.10 : Dummy questions [SOLVED]

[orion777]

I'm running gentoo 64 bit kernel 4.10 on Raspberry PI 3B (SAkakis' image viewtopic-p-8177442.html#8177442 ).
Now I wish to install MPTCP to get robust (redundant) network connection.
So, getting kernel sources and MPTCP patch for 4.10 into linux_mptcp:

Code: Select all

cd /usr/src/
git clone --depth 1 https://github.com/raspberrypi/linux.git -b rpi-4.10.y
cp linux linux_mptcp
cd linux_mptcp
make distclean
make bcmrpi3_defconfig
wget http://multipath-tcp.org/patches/mptcp-v4.10-d9c9f9273eb1.patch

Now applying the patch. It will apply with an error:

Code: Select all

patch -p1 < mptcp-v4.10-d9c9f9273eb1.patch
1 out of 46 hunks FAILED -- saving rejects to file net/ipv4/tcp_output.c.rej
	pi64 /usr/src/linux_mptcp # cat net/ipv4/tcp_output.c.rej
	--- net/ipv4/tcp_output.c
	+++ net/ipv4/tcp_output.c
	@@ -1282,7 +1307,7 @@ int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len,
	  * eventually). The difference is that pulled data not copied, but
	  * immediately discarded.
	  */
	-static void __pskb_trim_head(struct sk_buff *skb, int len)
	+void __pskb_trim_head(struct sk_buff *skb, int len)
	 {
	        struct skb_shared_info *shinfo;
	        int i, k, eat;

Patch was prepared for the vanilla 4.10 kernel; however gentoo 4.10 kernel have different string https://github.com/torvalds/linux/commi ... 3c3e9fc805
So now we have to manually change static int __pskb_trim_head(struct sk_buff *skb, int len) to int __pskb_trim_head(struct sk_buff *skb, int len) (changing void to int, because of commit https://github.com/torvalds/linux/commi ... 3c3e9fc805 ))

Now kernel make will fails, as ./include/net/tcp_output.c was changed by the patch to vanilla string. Ok, lets edit it manually for the gentoo:

Code: Select all

cd ./include/net/
edit by MC tcp_output.c
void __pskb_trim_head(struct sk_buff *skb, int len);
	to
	int __pskb_trim_head(struct sk_buff *skb, int len);

Now kernel can be maked without fatal errors. So its time to configure .config https://wiki.gentoo.org/wiki/MultiPath_TCP
However, this how-to seems not full.. I was truing it, but later I was adding more in .config to specify MPTCP "defaults":

Code: Select all

make menuconfig
¦     -> Networking support (NET [=y])                                    ¦
  ¦       -> Networking options                                             ¦
  ¦         -> TCP/IP networking (INET [=y])                                ¦
  ¦ (1)       -> The IPv6 protocol (IPV6 [=m]) to KERNEL (or DISABLE, but my is in kernel..)

then

Networking support (NET [=y])
  Networking options
    TCP/IP networking (INET [=y])
      [*]   IP: advanced router
        [*]     IP: policy routing
      [*]   MPTCP protocol
      [*]     MPTCP: advanced path-manager control  --->
        <*>   MPTCP Full-Mesh Path-Manager
        <*>   MPTCP Linked Increase
	 Default MPTCP Path-Manager (Full mesh)  --->  Full mesh
	MPTCP: advanced scheduler control                            x x
         <*>   MPTCP Round-Robin                                          x x
  x x    <*>   MPTCP Redundant                                            x x
  x x          Default MPTCP Scheduler (Redundant)  --->   

and

Networking support (NET [=y])
  Networking options
    TCP/IP networking (INET [=y])
      [*]   TCP: advanced congestion control  --->
        <*>   MPTCP COUPLED CONGESTION CONTROL (!!Can't be found in my menuconfig!!)
        <*>   MPTCP Opportunistic Linked Increase
	        Default TCP congestion control (Lia)  ---> Lia

Now it is possible to make the kernel:

Code: Select all

make
make modules_install
make firmware_install
cp -v arch/arm64/boot/Image /boot/kernel8.img
cp -v arch/arm64/boot/dts/broadcom/bcm{2710,2837}-rpi-3-b.dtb /boot/
sync
reboot

Ok, now its time to set MPTCP settings, as is shown here http://multipath-tcp.org/pmwiki.php/Use ... ureRouting
We have two network cards (the first one is build-in ethernet, the second one is virtual ethernet 3G/4G dongle (cdc-ethernet, sometimes called "stick")).

Code: Select all

pi64 ~ # ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.118  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::5102:cac7:da40:234d  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:28:f1:db  txqueuelen 1000  (Ethernet)
        RX packets 961  bytes 68443 (66.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 470  bytes 63405 (61.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi64 ~ # ifconfig eth3
eth3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.8.10  netmask 255.255.255.0  broadcast 192.168.8.255
        inet6 fe80::5d49:fee0:c509:bb11  prefixlen 64  scopeid 0x20<link>
        ether 0c:5b:8f:27:9a:64  txqueuelen 1000  (Ethernet)
        RX packets 60  bytes 25286 (24.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18  bytes 1710 (1.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Both obtain its ip by dhcp.
Lets check them:

Code: Select all

pi64 ~ # ping -I eth0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.1.118 eth0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=21.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=23.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=18.5 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 18.533/21.011/23.060/1.880 ms
pi64 ~ # ping -I eth3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.8.10 eth3: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=655 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=53 time=635 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=53 time=75.4 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 75.436/455.283/655.183/268.715 ms
pi64 ~ #

Ok, both are working. Now it is time to configure http://multipath-tcp.org/pmwiki.php/Use ... ureRouting

pi64 ~ # # This creates two different routing tables, that we use based on the source-address.
pi64 ~ # ip rule add from 192.168.1.118 table 1
pi64 ~ # ip rule add from 192.168.8.10 table 2
pi64 ~ # # Configure the two different routing tables
pi64 ~ # ip route add 192.168.1.0/24 dev eth0 scope link table 1
pi64 ~ # ip route add default via 192.168.1.1 dev eth0 table 1
pi64 ~ # ip route add 192.168.8.0/24 dev eth3 scope link table 2
pi64 ~ # ip route add default via 192.168.8.1 dev eth3 table 2
pi64 ~ # # default route for the selection process of normal internet-traffic
pi64 ~ # ip route add default scope global nexthop via 192.168.1.1 dev eth0

Now lets check resulting settings, as specified here http://multipath-tcp.org/pmwiki.php/Use ... ureRouting

Code: Select all

pi64 ~ # ip rule show
0:      from all lookup local
32764:  from 192.168.8.10 lookup 2
32765:  from 192.168.1.118 lookup 1
32766:  from all lookup main
32767:  from all lookup default
pi64 ~ # ip route
default via 192.168.1.1 dev eth0
default via 192.168.1.1 dev eth0 src 192.168.1.118 metric 202
default via 192.168.8.1 dev eth3 src 192.168.8.10 metric 204
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.118 metric 202
192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.10 metric 204
pi64 ~ # ip route show table 1
default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 scope link
pi64 ~ # ip route show table 2
default via 192.168.8.1 dev eth3
192.168.8.0/24 dev eth3 scope link

So Ok.. @blind@ configuring was done (blind because I dont understand what I'm doing...)

Now lets check network connectivity. Due of setting Default MPTCP Scheduler (Redundant) in the kernel, I suppose that pings' ICMP packet will be sent thru both networks, so I will widrawn eth0 network cable during the experiment and will expectate that connectivity will be still available:

Code: Select all

 ping -O 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=20.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=19.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=55 time=19.9 ms
no answer yet for icmp_seq=4
no answer yet for icmp_seq=5
no answer yet for icmp_seq=6

So, as can be seen, as soon eth0 cable was interrupter (AFTER its network router, that why we do not see From 192.168.1.1 icmp_seq=18 Destination Net Unreachable) the data transmission also aborts.

NOTE that network cards are still accessible directly:

Code: Select all

pi64 ~ # ping -I eth0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.1.118 eth0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=20.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=55 time=17.9 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 17.964/19.376/20.789/1.419 ms
pi64 ~ # ping -I eth3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.8.10 eth3: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=660 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=53 time=349 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 2 received, 33.3333% packet loss, time 2002ms
rtt min/avg/max/mdev = 349.341/504.981/660.622/155.642 ms

So, I don't understand WHAT to DO to make redundant connection, based on two network cards. Seems that MPTCP is not working in my setup
If someone have any experience working with MPTCP, please suggest something..
(I'm not experience user in linux, so some typical mistakes are also possible)

[orion777]

So, as I understood, nobody was/is using MPTCP on gentoo?

[Hu]

Nobody satisfies all of:

1. Reads and posts on the forums
2. Uses MPTCP on Gentoo
3. Can leverage #2 to help you

There might be MPTCP users who do not read the forums and have not seen this thread, or who use it, but lack the expertise to assist you.

[Ant P.]

Is there anything you need MPTCP for that you can't do using the simpler, built in multipath ip routing?

[orion777]

Initially I was planning to use NetworkBonding to be able to send same packet thru different network cards simultaneously.
Due of the fact that I'm not experience user, I spend two months to get network configuration running (to understand interrelationship between NetworkManager, dhcpcd, dhclient and what is written in gentoo tutorial https://wiki.gentoo.org/wiki/Handbook:X ... Networking ). And ONLY after that I understand that Network Bonding cannot be build using different network providers... (it was discussed and I was getting the answer here viewtopic-t-1074802.html . Also there is an advice to use two VPN connections to be able to bond them togehter, but I was not truing this solution yet)

So, to be able to send same packet over two different networks (in fact, two 3G/4G modems that emulates virtual ethernet cards) simultaneously I try to use mentioned here viewtopic-t-1060694-start-0.html MPTCP solution. But due of the fact that I'm dummy, I cant force it to work

What You means under built in multipath ip routing ?

[Ant P.]

Enaable CONFIG_IP_ROUTE_MULTIPATH and the kernel will automatically spread packets among routes that point to the same destination.

[orion777]

Do You mean this?

Code: Select all

 Symbol: IP_ROUTE_MULTIPATH [=y]                                         
  Type  : boolean                                                         
  Prompt: IP: equal cost multipath                                        
    Location:                                                             
      -> Networking support (NET [=y])                                    
        -> Networking options                                             
          -> TCP/IP networking (INET [=y])                                
  (4)       -> IP: advanced router (IP_ADVANCED_ROUTER [=y])              
    Defined at net/ipv4/Kconfig:81                                        
    Depends on: NET [=y] && INET [=y] && IP_ADVANCED_ROUTER [=y] 

As can be seen, it is enabled in my kernel. Also I have iproute2 4.14.1 installed.
What then I have to do to be able to build redundant connection (that is sending packets over two interfaces in parallel)? This one?

<p><b>Использование 2 линков</b>:<br>
В ядре: CONFIG_IP_ADVANCED_ROUTER=y, CONFIG_IP_MULTIPLE_TABLES=y,
CONFIG_IP_ROUTE_MULTIPATH=y<br>
<br>
# Настройка первого линка<br>
ip rule add from 63.89.102.157 lookup 1<br>
ip route add 10.0.0.0/24 via 10.0.0.1 table 1<br>
ip route add 0/0 via 63.89.102.1 table 1<br>
<br>
# Настройка второго линка<br>
ip rule add from 65.3.17.133 lookup 2<br>
ip route add 10.0.0.0/24 via 10.0.0.1 table 2<br>
ip route add 0/0 via 65.3.17.1 table 2<br>
<br>
Для проверки результата наберите: ip route list [table 1]<br>
<br>
<p><b>Балансиковка трафика</b>:<br>
ip route add default equalize<br>
nexthop via 63.89.102.1 dev eth1<br>
nexthop via 65.3.17.1 dev eth2<br>

[bbgermany]

Hi,

bonding of wan connections is possible, but your provider MUST support this. Usually they dont. Did you have a look at this: http://lartc.org/howto/lartc.rpdb.multiple-links.html

Since you are trying to "load balance" connections via your two 3G/4G interfaces, this could help you.

greets, bb

[orion777]

bonding of wan connections is possible, but your provider MUST support this. Usually they dont. Did you have a look at this: http://lartc.org/howto/lartc.rpdb.multiple-links.html
Since you are trying to "load balance" connections via your two 3G/4G interfaces, this could help you.

I'm mostly looking for the solution how to get fail-save operation (it is necessary to build network solution for the remotely piloted robot, whet one provider may not have sufficient coverage, signal strength, etc.. that why I wish to use two operators simultaneously).
In the how-to mentioned above I cant understand what means Finally, let $P1_NET be the IP network $P1 is in,
$IF1 - interface name, like eth0
$IP1 - interface IP, like 192.168.0.10
$P1 - gateway IP, like 192.168.0.1
$P1_NET - ?? net mask? 192.168.0.10/24 ? or something like 192.168.0.0 ?

[bbgermany]

Thats quite easy:

if you get the ip address 192.168.0.1 ($IP1) from your provider on your first 3G/4G interface ($IF1). The fitting network ($P1_NET) could be 192.168.0.0. Gateways and mask and even the subnet is generated by the provider. You have to look it up via ifconfig.

greets, bb

[orion777]

So, here is my existing configs "by default"

Code: Select all

 # ip route show
default via 192.168.1.1 dev eth0 src 192.168.1.118 metric 202
default via 192.168.8.1 dev eth3 src 192.168.8.10 metric 204
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.118 metric 202
192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.10 metric 204

So, as is specified here, at first we have to create two network tables, called T1 and T2.
I we found solution HOW to do this in http://www.allgoodbits.org/articles/view/24 in the section Create a custom routing table: echo "1 myorg" >> /etc/iproute2/rt_tables

Then configuration has to be applied. So:

Code: Select all

echo "T1" >> /etc/iproute2/rt_tables
echo "T2" >> /etc/iproute2/rt_tables

ip route add $192.168.1.0 dev eth0 src 192.168.1.118 table T1
ip route add default via 192.168.1.1 table T1
ip route add 192.168.8.0 dev eth3 src 192.168.8.10 table T2
ip route add default via 192.168.8.1 table T2

ip route add 192.168.1.0 dev eth0 src 192.168.1.118
ip route add 192.168.8.0 dev eth3 src 192.168.8.10
ip route add default via 192.168.1.1
ip rule add from 192.168.1.118 table T1
ip rule add from 192.168.8.10 table T2

ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1
nexthop via 192.168.8.1 dev eth3 weight 1

However, whet two T1 and T2 has been added, nothing can be writen inside into it:

Code: Select all

# ip route add $192.168.1.0 dev eth0 src 192.168.1.118 table T1
Database /etc/iproute2/rt_tables is corrupted at T1

Error: argument "T1" is wrong: "table" value is invalid

Code: Select all

pi64 /etc/iproute2 # cat rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
T1
T2

[bbgermany]

I think you did it wrong a bit; You need to add a number for your routing as well:

Code: Select all

echo "1   myorg" >> /etc/iproute2/rt_tables

Not only:

Code: Select all

echo "myorg" >> /etc/iproute2/rt_tables

greets, bb

[orion777]

Yes, number and name, separated by "TAB", like echo "1 T1" >> /etc/iproute2/1rt_tables
But now I have two more problems..

Code: Select all

# ip route add $192.168.1.0 dev eth0 src 192.168.1.118 table T1
# ip route add default via 192.168.1.1 table T1
# ip route add 192.168.8.0 dev eth3 src 192.168.8.10 table T2
# ip route add default via 192.168.8.1 table T2
# ip route add 192.168.1.0 dev eth0 src 192.168.1.118
# ip route add 192.168.8.0 dev eth3 src 192.168.8.10
# ip route add default via 192.168.1.1
# ip rule add from 192.168.1.118 table T1
# ip rule add from 192.168.8.10 table T2
# ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1
RTNETLINK answers: File exists
# nexthop via 192.168.8.1 dev eth3 weight 1
-bash: nexthop: command not found

[bbgermany]

Code: Select all

ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1  nexthop via 192.168.8.1 dev eth3 weight 1 

thats one command not two.

greets, bb

[orion777]

thats one command not two.

Ok... However, now I get:

Code: Select all

ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1 nexthop via 192.168.8.1 dev eth3 weight 1
RTNETLINK answers: File exists

So, as was specified here for some similar problem, I was replacing ip route add by ip route replace , so now it looks like this:

Code: Select all

ip route replace $192.168.1.0 dev eth0 src 192.168.1.118 table T1
ip route replace default via 192.168.1.1 table T1
ip route replace 192.168.8.0 dev eth3 src 192.168.8.10 table T2
ip route replace default via 192.168.8.1 table T2
ip route replace 192.168.1.0 dev eth0 src 192.168.1.118
ip route replace 192.168.8.0 dev eth3 src 192.168.8.10
ip route replace default via 192.168.1.1
ip rule add from 192.168.1.118 table T1
ip rule add from 192.168.8.10 table T2
ip route replace default scope global nexthop via 192.168.1.1 dev eth0 weight 1 nexthop via 192.168.8.1 dev eth3 weight 1
 

Now ping, as well any wget is done thru eth3 only
Yes, metric is different.. So maybe this solution is applicable to same network cards only?

Code: Select all

pi64 ~ # ip route
default
        nexthop via 192.168.1.1 dev eth0 weight 1
        nexthop via 192.168.8.1 dev eth3 weight 1
default via 192.168.1.1 dev eth0 src 192.168.1.118 metric 202
default via 192.168.8.1 dev eth3 src 192.168.8.10 metric 204
192.168.1.0 dev eth0 scope link src 192.168.1.118
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.118 metric 202
192.168.8.0 dev eth3 scope link src 192.168.8.10
192.168.8.0/24 dev eth3 proto kernel scope link src 192.168.8.10 metric 204

[bbgermany]

Im very sorry, but I cannot help you with this, since I connot test it by myself. But I think, it could be related to the default gateways, your provider gives you, when you start up your connections.

greets, bb

[orion777]

I have two suggestions:
1) this solution cant work with network cards with different metrics
2) Both my cards gets IP by means of DHCP (dhclient, dhcpcd, NetworkManager). NetworkManager (as I understand) writes default gateway.. So maybe I cant override this default gateway settings?

[orion777]

I have an answer.. and it is very simple! I found this in "options->default" description: 'default': This scheduler is the default one. It will first send data on subflows with the lowest RTT until their congestion-window is full. Then, it will start transmitting on the subflows with the next higher RTT. So, I can conclude that MPTCP uses TCP window as a measurement parameter to able to work. This means that MPTCP can work only with TCP. However, in my testing, I was using "ping", so ICMP packets. As I know, ICMP packets does not use windows, so maybe that why all my MPTCP tests were failed. Then I was trying to initiate TCP/IP data flow by wget something and both cards starts to work; also if one card is failed, down, on lost connection and stay live, second card takes all load automatically.

A few words about settings.
- First, you have to patch your kernel.
- Then it is necessary to enter additional routing settings. This can be done manually each time after system reboot or can be automated by various scripts (suggested gentoo script has mistakes and will not work. If someone know, please suggest me how to fix it ). All related info (manual and auto scripting) is here
- Default settings are specified during kernel make.
- Settings, those are set by default in kernel, can be overridden manually in live system thru sysctl
--- It is recommended to select default. In this mode MPTCP will use both network cards (not only one!) and your speed of connection will be doubled. Traffic balancing is fully flexible, based on
TCP window (as soon it is full - traffic is sent to the second network card and etc). Round-robin is not recommended because it is experimental,... seems that it has no flexibility and can work fine only with equal speed networks.