Openssl performance issues

[jesnow]

I am using a lot of ssh these days via ssh tunnels, but the performance is pretty bad. My new rabbit hole is to make sure that I have a correct ssl setup and that it's performing optimally. I'm writing down what I found in case it is useful later. Please criticize me in this thread if I make any mistakes, and please help if you have any ideas about how to troubleshoot my rotten throughput. I'm tunneling from my place of work (workstations vanaert and pogacar) to my home (bartali and merckx). All have wired ethernet and get ~1Gb/s conneciton speeds to the internet. It's not that easy it turns out to find out what's really going on because there aren't many introductory guides for people like me. Skip to the end for a description of the problems I've been getting.

Introduction

Openssl is a robust, full-featured Open Source toolkit for the Transport Layer Security (TLS). Openssl uses cryptographic engines to implement ciphers that exchange certificates to establish encrypted connections outside and inside the operating system instance.

https://www.openssl.org/
https://packages.gentoo.org/packages/dev-libs/openssl

The number of gentoo packages that depend on openssl is impressive, and it's sometimes surprising what all uses it. It's really a fundamental part of the linux operating system and gentoo in particular. Fortunately, it's pretty trouble-free. The default implementation included in gentoo "just works" and if you follow the gentoo installation guide, you have a solid engine and plenty of available ciphers of excellent quality to work with. Most people can stop reading right here.

Openssl vs libressl

Those of us old enough remember that there was a controversy, a fork, many flame wars, and a confusing situation for users. For a while there was a choice of which TLS implementation to use, openssl or libressl. This is a feature of open source software, not a bug. Many electrons have been spilled over this. But this particular competition is now resolved in favor of openssl, it's a long story:

https://wiki.gentoo.org/wiki/LibreSSL

ssl vs ssh

Most of us knowingly only use ssl by using ssh to log into another machine. ssl does much much more, it's the underlying technology, also used by a lot of other communication protocols, but still the best way to test it in the real world is through ssh. You should have public key encryption set up in ssh, at the very least generated keys installed for localhost. How to do that is beyond the scope of this article, there are a million guides out there.

https://wiki.gentoo.org/wiki/SSH

Versions

Openssl is under steady development, but the changes are intended to be transparent to the user, and only really relevant to developers. Currently in tree are:

[Genone]

First, your tunnel test is only using a 1MB sample size, so any delay caused by connection setup/teardown will have a much greater effect than in your other tests using a 100MB / 200MB sample size.
Also your tests show that the bottleneck is not openssl itself, but rather an issue specific to the tunneling setup, so the title is a bit misleading. More so given that you've omitted any information about how you setup your tunnels. Like using an ssh connection over an ssh tunnel obviously won't work very well as you're encrypting/decrypting the data twice on both ends for no reason (though I'm not an expert on ssh tunnels, so that could be wrong).

What is really odd is that the 256 bit ciphers in your tunnel test seem to perform much better than the 128 bit variants. Should really retest with a much larger sample to ensure you're actually looking at bandwidth rather than (random) latency. As a general rule: high-level performance tests should run for more than just milliseconds, esp. if network connections are involved.
Or just drop the |grep|awk part and look at the actual times. Also look at actual CPU usage during your tests, esp. if it changes significantly between different tests.

Last but not least, ssh has a -v option for diagnostics that might be useful to you.

[no101]

You might be experiencing "TCP Meltdown". Running TCP over TCP can cause problems. Wireguard uses UDP for exactly this reason and OpenVPN suggests using UDP as well. VPN is an inversion of what you're doing but the issue is the same.

I suspect tcp meltdown because the localhost network is perfect: you never get dropped packets or congestion. Since you never get errors, localhost is pretty useless for testing network code performance. There's other differences too like you will likely get zero-copy network transmission because the kernel knows it can simply pass the existing buffer.

I don't really have a good reference but OpenVPN has a simple explanation in their FAQ.
https://openvpn.net/faq/what-is-tcp-meltdown/

[szatox]

I just want to add that in my experience SSH makes a very slow data pipe. Even an otherwise idle system wouldn't transfer more than 20MBps per connection.
Fortunately, in my particular case I could simply work around this by splitting a job into a few thousand chunks and sending up to 20 of those in parallel for a total bandwidth of 400MBps but I still do not consider it a good general-purpose solution.

Just saying. If you do have a full control over both ends, you'll be better of simply shoving your data down netcat wrapped in wireguard.
Anyway, what do you need it for? I wonder what other options are available.

[jesnow]

To summarize:

I've got networking running very reliably on multiple machines outside my home network using dynamic dns, a pinhole in my router firewall on a random port, ssh reverse tunneling (using openssh) and cifs. The nice thing is it all works about the same as it does at home, and my home systems can connect easily to my remote systems at work through the tunnel. It's like a simple vpn.

Here's my remote machine .ssh/config: