MPTCP on RPI4B with armbian + patched rpi-sources

[szatox]

Long story short, I wanted to enable MPTCP on my RPI4B with armbian.

I've been using it successfully on AMD64 via a kernel patch that attempts to upgrade all TCP sockets to MPTCP. If MPTCP is not supported by the remote end, kernel automagically fails back to TCP anyway, so there is no harm carpet-bombing it like that. Anyway, I though it would be a good idea to apply the same patch to my rpi. [post]8826211[/post]

I crosscompiled raspberrypi-sources according to https://www.raspberrypi.com/documentati ... ernel.html and transplanted it into previously installed armbian I've been using for a while. It boots, services work, I even recompiled an out-of-tree wifi module to get a USB dongle back. So far so good... But mptcp doesn't work.
Tried again with IPv6 permanently built into kernel instead of as a module, enabled MPTCP for IPv6, and it still doesn't activate.
The only difference compared to earlier is that when I ssh to it from a laptop where MPTCP actually works, I see 2 of those lines instead of 1:

setsockopt socket 3 IPV6_TCLASS 72: Operation not supported

It still connects though.
Connecting via ipv4 doesn't print any errors at all. In either case, the session does not transfer between ethernet and wifi like expected (It does work with my VPS; I can switch between wifi and a cellphone modem without dropping SSH session; RPI has multiple interfaces and VPS only 1, but AFAIK it shouldn't matter)

There are no firewalls, independently of each other both links work, it's just not using multipath.

Code: Select all

rpi # sysctl -a | grep mptcp
net.ipv4.tcp_available_ulp = mptcp
net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.checksum_enabled = 0
net.mptcp.enabled = 1
net.mptcp.pm_type = 0
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4

Code: Select all

laptop  # sysctl -a | grep mptcp
net.ipv4.tcp_available_ulp = mptcp
net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.available_schedulers = default
net.mptcp.blackhole_timeout = 3600
net.mptcp.checksum_enabled = 0
net.mptcp.close_timeout = 60
net.mptcp.enabled = 1
net.mptcp.pm_type = 0
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4

So, any ideas?


Edit: I snooped on the traffic with Wireshark, it does recognize the conversation as an MPTCP stream, however there is only 1 substream with index 0. I'm pretty sure endpoints should advertise their other IPs, so maybe I need to play with routing rules a bit to get them to advertise the other IPs....

[szatox]

So, I did some tests and digging through docs, and it seems mptcp just really doesn't like local addresses. Which kinda makes sense, but not completely, and regardless of the merit, is quite annoying. And I'm currently not sure what to do about it.

Interestingly, nc in client mode somehow evades my kernel-level socket upgrade. On the listening side it opens an mptcp socket as expected, but I had to find an alternative client.

[szatox]

Those are pretty useful links in case anyone wanted to play with mptcp:
https://developers.redhat.com/articles/ ... he_testbed
https://developers.redhat.com/blog/2020 ... 1-subflows#

Unfortunately, it doesn't really explain conditions required for it to work.
It does work in the "testbed", using private IPs, but 2 namespaces connected to each other and nothing else is not a very useful network.
I connect form my laptop to the internet. I connected from rpi to my laptop (over LAN). I can't connect from laptop to rpi - a plain, simple tcp connection is initiated instead.

A connection can be flagged as mptcp even when interfaces don't have any mptcp flags set.
Redundant routes are not necessary either for starting mptcp session, though in this case connection will obviously be stuck with a single subflow.
I guess I'll have to simplify my network configuration first and try again, enabling one thing and a time until mptcp on LAN breaks again.

[szatox]

Well, I still have some routing to do there, but the immediate problem of should-be-mptcp-starting-as-plain-tcp went away after reboot.
I got to the point, where I can switch networks 2-3 times before connection breaks, which seems to be a problem with mptcpd removing endpoints on link-down which in turn completely unnecessarily breaks the related subflow related to the same address.
Not that mptcpd is necessary in the first place, I think I can work around it with a shell script.

Or doing some static configuration. Which I knew would actually be the easiest solution in my particular case... But checking out other options was more interesting.