wan performance sucks: raw vs ssh tunnel vs wireguard

jesnow · l33t Joined: 26 Apr 2006 Posts: 856

Hi everybody:

I previously posted about setting up wireguard, that's now wrapped up.

https://forums.gentoo.org/viewtopic-t-1161123-highlight-.html

My experience in practical use is still that the network performance over WAN is much much worse than in the LAN. It doesn't make sense. The standard answer is "what do you expect, wi-fi is slow". But there is no wi-fi involved here. I have wired ethernet all the way. My WAN ping is 30ms (that's supposed to be good), there's no reason for network performance overall to suck. I'm being methodical about it, and already discovered a bum ethernet card, which I replaced.

I next want to show my testing of pure network performance using raw, ssh tunnel and wireguard tunnel methods. For those of you who don't want to read it all: Raw WAN connections are slower than connections on the LAN by only about 13%. Tunneling through ssh and wireguard is slower by about another 30-40%, and wireguard beats ssh head to head by about 35% in both transmit and receive. It's a very consistent and not so very surprising result, though I didn't expect wg to beat ssh by so much.

My setup is: local (pogacar) and remote (merckx) machines are both old-ish 3GHz core i7 machines with 16GB memory, connected by 1GBE to their respective providers. Ping is a consistent 30ms in both directions, and both get <1ms ping and 930 MB/s in their respective LANs. I have iperf3 server running on merckx. I run the iperf3 client on pogacar in both forward and reverse mode. So I can transfer data 6 ways: 1,2) by an open raw port in my router, 3,4) through an ssh tunnel (running on localhost:45201) on another open port, 5,6) through a wireguard tunnel in yet a third open public port.

This was a very simple test based on iperf3 default settings. I simply ran iperf3 in transmit and receive mode in both directions. I spent a few hours tuning various network parameters and found out, guess what: Linux has pretty good settings, don't mess with them. I could make network throughput dramatically *worse* without much effort, but never made a dent on improving either ssh or wg performance. So my recommendation is, don't mess with the default network settings. Smarter people than me created them.

Here are the command I ran

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2003

Out of interest, what is the WAN? Is it asymmetric? I only get 17 MB down and 800 KB up on my copper wires and ADSL, but you obviously have something faster. It may be worth logging on to your router/modem, if that's possible, and seeing what speeds it thinks the lines are.
_________________
Greybeard

jesnow · l33t Joined: 26 Apr 2006 Posts: 856

pingtoo · Posted: Sun Feb 19, 2023 7:12 pm Post subject:

Maybe it is my poor English comprehensive skill. But I don't get what is your expectation for WAN connection? Are you expecting it will be same or very very close to LAN speed?

I think in part of your seeking answer already lie in your initial post, you already state that ping response between WAN nodes is 30ms compare to LAN nodes 1ms.

In some way the "30ms" or "1ms" are known as latency.

In TCP/IP network usually we measure number of bytes transfers and number of packets transferred. Normal LAN ethernet packet size are 1500 (or less) so in order to transfer 1 MBytes of data you need proximate of 699 packets if the latency is 1ms, this mean it will take 699 * 1ms = 699 * 0,001 = 0.699 second to transmit the 1 MBytes of data.

However if the same 1 MBytes of data is to transmit over the WAN which have 30ms latency than it is 699 * 30ms = 699 * 0.03 = 20.97 seconds to transmit the 1 MBytes of data.
And usually WAN packet size varies from network to network so it is even harder to measure.

My above explain is purely illustration only. The try WAN is never simple as I posted above.

So it is very important if you want to increase transmission speed by trying to reduce the latency.

in TCP/IP network one of the factor that introduce latency is number of hops. A hop is defined as from one node to the next node on the same wire.

s0ulslack1 · n00b Joined: 06 Mar 2022 Posts: 20

Throughput is not Latency. You have 2 different locations over different providers going over multiple networks which include alot more hops vs your LAN.

deagol · n00b Joined: 12 Jul 2014 Posts: 61

jesnow · l33t Joined: 26 Apr 2006 Posts: 856

It's not that simple. By your reasoning no video streaming would ever be possible, but we stream HD video every day. Those protocols are optimized in such a way that latency doesn't matter, only throughput.

But you're asking for the larger context: I'm trying to mount filesystems over the internet so I can work anywhere in the world exactly like I do at home here. Those protocols are obviously *not* optimized to make latency unimportant. So that's why I'm going methodically to see if everything is functioning correctly at each level, and then seeing what I can change to improve my ability to work.

My conclusion in today's post is that tunneling protocols don't cost us very much throughput, and that wireguard is definitively faster than ssh. That's a step in the right direction.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3133

jesnow · l33t Joined: 26 Apr 2006 Posts: 856

Well what this definitely shows is that mt throughput is >50% of the wire speed, that's pretty good on an encrypted connection. What comes next is to see how well operating systems and application shove data through that bandwidth. the answer is: Horrors await.

Cheers,
Jon.

pingtoo · Posted: Sun Feb 19, 2023 11:01 pm Post subject:

I am glad the TCP window size brought in to conversation.

as matter of fact I am not able to answer where jesnow point, that stream HD video or every my own example of 1 MB transmit that take 20 some seconds. Because that not the reality. I admit there are a lot of factors in WAN that will affect performance.

In my post I just trying to point out WAN is a lot different than LAN. And there are a lot more factors that you have no control. And it usually display those factors blend in the ping response which I termed "latency".

I am not sure if there are any parameters for tuning Wireguard for window size (send/receive buffer) but as posted in this thread, many already point out the "latency" is the main concern and turning the window size will most likely give you the best bang on the buck.

I just want to give you my experience not to make the change of window size to global to everything that will running on the Wireguard peer nodes. I learn my lesson that if you did that you will run out of memory sooner than you expected and it is actually hard to understand when you have to debug in the production environment.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3133

Well, wireguard itself doesn't really need any window, since it uses UDP (and is stealthy), and it does not need to guarantee delivery, just like the plain copper wire itself does not guarantee it.
However, the point still stands: most of the data transferred will be sent using TCP (inside the tunnel).
So, wg should be able to operate at link-speed (if you have enough CPU power for the compression to meet the demand), but underlying WAN will always peak lower than underlying LAN of the same theoretical speed.

jesnow · l33t Joined: 26 Apr 2006 Posts: 856

I did all of this just with iperf, deciding that the raw network speed with wg was consistently better than ssh. I continued testing (I did a *lot* of it) using samba over both. Of course WAN speed is going to be less than LAN speed, and of course latency is the culprit. But It's a really really bad culprit, as it turns out. Much worse for file performance than it is for raw network pipe speed -- many orders of magnitude of slowdown! I did not expect that. I will start a new thread about this.

Most of the testing and tweaking posts about samba predate GBE becoming common in WANs.

More shortly.

Cheers,
jon.