Remote file access *slow*: smb, ssh, sshfs [SOLVED]

[jesnow]

Hi Everyone:

I have my gentoo boxes behind a firewall, as they should be. There is just a pinhole to the outside on a nonstandard port, and I listen on ssh for incoming connections. On the inside I'm running samba and it works quite well locally, with the usual complaints and workarounds.

Coming in from outside I am using a macbook pro with new catalina. That offered some problems, but also works. I can ssh in, I can sftp. Sshfs works but you have to remember to put the mount point in user-controlled space. The Mac won't let you use /Volumes as user. Same thing with reverse-tunneling samba -- you can't use privileged ports like 445 as user, even if you're not using them "Operation not permitted", well we knew this. The workaround with a 127.0.0.2 loopback alias didn't work either, that's new. No listening on low ports for user in Catalina, period. The new wrinkle is that medium ports between 1024 and (I think) 8192 don't work either. But they don't give an error message, they open but they don't work. So you do something like ssh me@myserver.net -L 127.0.0.1:44445:myserver.net:445 and you can mount the samba share in smb://me@myserver.net:44445/me, and off you go. Helps if you set up passwordless for the ssh login, it will still ask you for your samba password. All set, full access.

*** Here's the problem: It's slow as mud! Like 30 seconds to get the initial directory listing. This is true both for sshfs and for smb. Files transfer in both just fine, but directories take for-effing-ever to load.

The broadband connection is fast. The gentoo server (Core i7 16GB) has 1gbps up and down, My ping to where I am now is around 60ms, which I think is about as good as it gets, and d/l speed from the client 200mb/s, up only 5mb/s but still we dreamed about such transfer speeds back in the day. I can use zoom with no delay at all, even with ssh, in the terminal the directory listing is instantaneous.

I was thinking it was the finder previews that were holding things up, but I've noticed the directory displays without them, and only loads them later anyway.

Any insights you may have on how to get the whole thing to be more responsive, I would greatly appreciate.

Yes I have thought of alternatives to working this way, but I am trying to figure *this* part now and will go down the rdp rabbit hole later.

Cheers,

Jon.

[Hu]

If you use a Linux client and a Linux server, is it also bad? This would determine whether the problem is that the Linux server is always bad, or if there is something particular to how the OSX client behaves that is bad. Your subject line mentions ssh, but I don't see anything in the body about using ssh for its interactive shell mode, only for tunneling. When using ssh from OSX in this setup, how is the usability of the interactive shell? Is it also bad? Does it have high latency, poor throughput, or both? For example, if you run seq 1 1000000 (caution: prints 1 million lines!) over ssh, is the display smooth, or do the numbers arrive in batches? Likewise for sftp: you mention that it works, and do not specifically mention that it is unpleasant to use. If you transfer a file using both sftp and sshfs, is one substantially slower than the other? What version of ssh is used on each end? Historically, Apple has been very bad about shipping outdated third party programs, so it would be interesting to know whether their ssh client is even close to current.

[jesnow]

These are all good questions. I just got this laptop, and I'm in the boonies, so my ability to experiment with other machines is limited. I'm planning to dual boot this macbook pro, but that's in the future.

Using the interactive shell is smooth and instantaneous, like being there. I can run a text app that does a lot of updating like top, and it's hard to tell it's not in the local net. time seq 1 10000000 (note the seven zeroes) took 9.3 seconds and on the server xosview (running by x forwarding) shows 10M, I think that's MB/sec on, so 80 Mb/s actual. Considering the overhead I'd say the interactive shell is doing its job. Pure X forwarding is almost fast enough to use, thunderbird running on the home gentoo server for examples will run its display on my laptop's x server, but it's just a bit too laggy and slow.

On the home LAN I have the same setup, I use smb to serve all my work data to my other machines. That's why I thought once I got broadband here away from home it all ought to work pretty smoothly. It only seems to be getting the directory that's taking forever. Like something is putting a wait state in there that has to time out.

In fact, when I mount the drive, I can use xosview to watch the server, but you can't even see that something has happened. Xosview shows basically nothing happening for the 10-20 seconds the mac samba client is "loading". No bump in cpu, no bump in network traffic. File transfers are running about 40MB/s (again measured on the server) using the mac samba, that saturates the the download speed of my local bandwidth.

So shell good, file transfers good. Just walking the file system is what's super slow.

Can you think of other ways to test the various parts independently?

[update:] There is indeed a known issue with broken ssh in catalina 10.15.4, exactly for doing port forwarding. This isn't it, and I am using 10.15.5 on this mac, so it might be related.

Cheers,

Jon.

[Ant P.]

sshfs is pretty inefficient and likely to be CPU-bound, being both encrypted and implemented via FUSE. I don't know if SMB has the same problem but you might have better results with NFS.

[jesnow]

OK, so for the world to see: here's the solution for SMB:

You Have to configure ControlMaster, then all is good. Apparently it was authenticating for every connection, which may not matter whn yo open another terminal, but when you're getting a directory apparently...

There are several ways to do this, but I did this one, to put it in tmpfs:

in ~.ssh/config:

Code: Select all

host *
    controlmaster auto
    controlpath /tmp/ssh-%r@%h:%p

For the record: I was not processor bound, I'm pretty sure even an RPi could handle the full bandwidth of compression and encryption available on any normal cable connection. I have a recent macbook pro as a client, and at home an i7, so I'm pretty sure it was never bandwidth or processor issues.

Nfs is not a bad idea as I've heard it's less "chatty", but honestly, everything is running lickety split now, it seems. I will mark this "solved" and move on.

Cheers,

Jon.

[jesnow]

More mac relevant SMB speedups:

1) .DS_Store is your enemy.

You Mac SMB client reads all these files that store unnecessary attributes (Icon position in folder for example). You can delete them all off your server. AND disable using them by giving (in Terminal):

Code: Select all

$ defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE

This turns the whole business off.

2) vfs objects = fruit

What does this mean? who knows. It's gobbledygook that has to do with Apple's deviations from the smb protocol. But Samba can support them so better turn in on in smb.conf in the [global] section:

Code: Select all

 vfs objects = fruit 

From samba.org: "directory enumeration is enriched with Mac relevant filesystem metadata (UNIX mode, FinderInfo, resource fork size and effective permission), as a result the Mac client doesn't need to fetch this metadata individually per directory entry resulting in an often tremendous performance increase."

Needless to say this is what you want. More info at Samba.org on this.

3) net.inet.tcp.delayed_ack=0
Create edit the file /etc/sysctl.conf on the mac to include this line.

It has to do with (as the name implies) the handling of delayed TCP acknowledgements.

I wish I had found these things the first time around. All the googling I did, they didn't come up, so I'm including them here now.

Cheers,

Jon.

[Hu]

3) net.inet.tcp.delayed_ack=0

I cannot comment on whether this is good for Mac SMB performance, but I feel I should warn that TCP delayed acknowledgements are usually a good thing, so they should be disabled only when you have a specific problem (like this one) where the change produces an obvious improvement. The Wikipedia page also describes a case where they are undesirable, and this could be what caused the problem for you. Each side waited for the other to either send a packet or hit a timeout. No extra packets were sent, so the timeout had to occur before forward progress was made. If this happened repeatedly, it could quickly add up to a user-visible lag.