Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[solved] Rsync stalls while downloading
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sat Apr 26, 2014 9:10 pm    Post subject: [solved] Rsync stalls while downloading Reply with quote

I'm trying to rsync pull a series of archives from one remote system which is connected to a business DSL line to another remote system which is hosted in a data center. The rsync seems to stall at about the same point every time and I'm sure that both of the remote systems are online throughout the stall. Should I investigate further or contact the DSL ISP?

Last edited by grant123 on Tue May 06, 2014 9:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Sat Apr 26, 2014 10:25 pm    Post subject: Re: Rsync stalls while downloading Reply with quote

grant123 wrote:
The rsync seems to stall at about the same point every time and I'm sure that both of the remote systems are online throughout the stall. Should I investigate further or contact the DSL ISP?

grant123 ... is this 'rsync -e ssh' or not, and what version of rsync? I seem to remember a similar report (can't remember where exactly, perhaps here or b.g.o) of stalling. Anyhow, more information required ...

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sat Apr 26, 2014 10:29 pm    Post subject: Reply with quote

It is indeed 'rsync -e ssh' and rsync-3.0.9-r3 on both client and server.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Sun Apr 27, 2014 12:09 am    Post subject: Reply with quote

grant123 wrote:
It is indeed 'rsync -e ssh' and rsync-3.0.9-r3 on both client and server.

grant123 ... sorry, I should have also asked if the 'hpn' useflag is enabled (I'll assume yes as its the default with gentoo). So, the bug I was refering to above (ok, its old, and marked RESOLVED/FIXED but I don't know what your setup is, so it may be of some help) ... seems the solution (or at least something to try) is 'TcpRcvBufPoll no' in /etc/ssh/sshd_config (on the recieving/server end).

HTH & best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sun Apr 27, 2014 4:19 am    Post subject: Reply with quote

'TcpRcvBufPoll no' doesn't fix it unfortunately. I tried -hpn on the client and that didn't do it but I'll be able to try -hpn on the server tomorrow and I will report back.
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sun Apr 27, 2014 4:00 pm    Post subject: Reply with quote

I'll be testing with -hpn on the server shortly but I thought I'd mention that I can push to the server without any problem. The issue only manifests itself when pulling from the server. I also tested with scp with the same problem.
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Thu May 01, 2014 2:55 pm    Post subject: Reply with quote

Unfortunately -hpn on both the client and the server do not fix the stall. Any other ideas?
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Thu May 01, 2014 5:50 pm    Post subject: Reply with quote

What would be the easiest way to test file transfer without openssh? I don't currently run a webserver on the backup server.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Thu May 01, 2014 9:56 pm    Post subject: Reply with quote

grant123 wrote:
Unfortunately -hpn on both the client and the server do not fix the stall. Any other ideas?

grant123 ... well, other than parsing the logs (particularly if iptables is in use) of server, client, and gateways (if possible) then no, not really, you might try and disable compression on the client end (but I doubt this is the issue) ...

~/.ssh/config
Code:
Host foo
  Hostname foo.domain.tld
  Compression no

grant123 wrote:
What would be the easiest way to test file transfer without openssh? I don't currently run a webserver on the backup server.

probably 'netcat' (here using 'ncat' from net-analyzer/nmap[ncat] ... though there are (many) other netcat's)

Code:
# ncat -v -l -p 80 < file

... and the recieveing end ...

Code:
# ncat foo.domain.tld 80 > file

HTH & best ... khay
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Fri May 02, 2014 11:16 am    Post subject: Reply with quote

khayyam wrote:
[...] you might try and disable compression on the client end (but I doubt this is the issue) ...

correction ... that should be "enable" not "disable" as per comment 29 in the above bug.

~/.ssh/config
Code:
Host foo
  Hostname foo.domain.tld
  Compression yes

... sorry for any confusion ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sat May 03, 2014 2:44 am    Post subject: Reply with quote

Wow, I'm surprised to find that even though rsync consistently fails to pull 50MB without stalling, netcat transferred that amount successfully the first time. So there's an SSH pull problem (pushing works fine). I've compiled SSH without HPN on both the client and server and I've enabled compression on the client with -C, but the rsync transfer still stalls. Where else should I look?
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Sat May 03, 2014 12:46 pm    Post subject: Reply with quote

grant123 wrote:
Wow, I'm surprised to find that even though rsync consistently fails to pull 50MB without stalling, netcat transferred that amount successfully the first time. So there's an SSH pull problem (pushing works fine).

grant123 ... the strange thing is this has all the hallmarks of the HPN issue, you did restart sshd on updating (and logged out and back in so that the ssh session was not using some HPN enabled process ... if say 'ForwardAgent yes' was enabled)?

grant123 wrote:
I've compiled SSH without HPN on both the client and server

I should have stated that it probably wasn't necessary to re-emerge with -hpn as hpn can be disabled via "HPNDisabled yes" in /etc/ssh/sshd_config.

grant123 wrote:
[...] and I've enabled compression on the client with -C, but the rsync transfer still stalls. Where else should I look?

I would probably try the transfer using '-vvv' (maximum verbose) and see if anything is reported when it stalls, I would also look at any other possible contributing factors (kernel version, firewall, etc) most importantly (as stated above) log files.

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sun May 04, 2014 3:44 am    Post subject: Reply with quote

Quote:
the strange thing is this has all the hallmarks of the HPN issue, you did restart sshd on updating (and logged out and back in so that the ssh session was not using some HPN enabled process ... if say 'ForwardAgent yes' was enabled)?


Yes, I've restarted the server and client since then and I tested again today with the same result.

Quote:
I should have stated that it probably wasn't necessary to re-emerge with -hpn as hpn can be disabled via "HPNDisabled yes" in /etc/ssh/sshd_config.


I tried setting HPNDisabled but it was unrecognized which I think confirms that my install does not include HPN.

Quote:
I would probably try the transfer using '-vvv' (maximum verbose) and see if anything is reported when it stalls, I would also look at any other possible contributing factors (kernel version, firewall, etc) most importantly (as stated above) log files.


Upgraded to 3.14.2 with the same result and -vvv was not enlightening unfortunately.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Sun May 04, 2014 11:47 am    Post subject: Reply with quote

grant123 wrote:
I tried setting HPNDisabled but it was unrecognized which I think confirms that my install does not include HPN.

grant123 ... yes, obviously, what I ment was there was actually no need to re-merge with -hpn, it could have been toggled via HPNDisabled.

Anyhow, I wonder if this isn't a MTU/fragmentation problem ... try setting the mtu to 576 as suggested.

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Sun May 04, 2014 4:06 pm    Post subject: Reply with quote

Quote:
I wonder if this isn't a MTU/fragmentation problem ... try setting the mtu to 576 as suggested.


You nailed it! I changed both interfaces to MTU 576 on the server and client and the problem disappeared. I suspect the root of the problem is that I'm using an AT&T modem/router with the server. I've had trouble using a proxy server there before and I discovered that the attached modem/router doesn't send ICMP responses. The solution proposed by AT&T was to put the modem/router into bridged mode but it's remote so I haven't been able to do that.

Could this MTU discovery also point to the modem/router and could putting it into bridged mode solve the problem? What can I do in the meantime?

Is it strange that netcat works with MTU 1500 and openssh doesn't?
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Sun May 04, 2014 9:24 pm    Post subject: Reply with quote

grant123 wrote:
Quote:
I wonder if this isn't a MTU/fragmentation problem ... try setting the mtu to 576 as suggested.

You nailed it! I changed both interfaces to MTU 576 on the server and client and the problem disappeared.

grant123 ... ok, good.

grant123 wrote:
I suspect the root of the problem is that I'm using an AT&T modem/router with the server. I've had trouble using a proxy server there before and I discovered that the attached modem/router doesn't send ICMP responses. The solution proposed by AT&T was to put the modem/router into bridged mode but it's remote so I haven't been able to do that.

You could probably put dd-wrt or open-wrt on it (see: list of router or firewall distributions for other possible options) ... but this might violate your terms of service. As for bridging, well, yes if you have a second device behind it (basically, putting it in bridge mode will turn it from a modem/router into a modem).

grant123 wrote:
Could this MTU discovery also point to the modem/router and could putting it into bridged mode solve the problem?

I can't say for sure if its the router or not, its the most likely culprit however. As for solving the problem, well, yes, if your aching feet can be solved by cutting your legs off :) ... basically, you're just disabling its ability to do routing.

grant123 wrote:
What can I do in the meantime?

Well, you can set the mtu in /etc/conf.d/net ...

Code:
mtu_eth0="576"

Otherwise, find a better firmware for the device or replace it with something else. Only the other day I found a Comtrend CT-6373 wireless router (4x gigabit ethernet, 1x a/b/g wireless, 8mb flash, 32mb RAM, USB) that's supported by open-wrt, I haven't as yet installed anything on it but once I do it'll be able to do the equivalent of a fairly decent Cisco router/WAP. So, just look around, there is a ton of this kind of HW thrown out every day (as ISPs tend to bundle them with the ADSL package and when some feature is upgraded ... such as 802.11g to 802.11n ... they just get thrown out).

grant123 wrote:
Is it strange that netcat works with MTU 1500 and openssh doesn't?

ummm ... doesn't that link I provided explain why it effects ssh?

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Mon May 05, 2014 8:38 pm    Post subject: Reply with quote

Quote:
As for bridging, well, yes if you have a second device behind it (basically, putting it in bridge mode will turn it from a modem/router into a modem).

I have a Gentoo (laptop) router behind it so I think that's what I'll do.

Quote:
doesn't that link I provided explain why it effects ssh?

I just read it again but I'm not seeing the answer. Could you give me a clue?

That's a lot for all your help.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Mon May 05, 2014 9:20 pm    Post subject: Reply with quote

grant123 wrote:
Quote:
doesn't that link I provided explain why it effects ssh?

I just read it again but I'm not seeing the answer. Could you give me a clue?

grant123 ... I would if I could remember. I didn't happen to read the link I provided, I just remembered the MTU issue with ssh from some years back and that link looked to explain it. As I remember this also effects SSL (perhaps the reason your proxy had issues) and basically it boils down to encrypted protocols being bad at handling packet fragmentation (by design and for good reason) the issue comes about due to the overhead of datagram encapsulation (as the packet passes between routers its added to) so (again, as I remember) as the packet is marked "don't fragment" the increase in size passes above the MTU and it stalls.

Why does this not effect everyone? Well, I'm not sure exactly, I assume its due to some routes being less congested than others, and/or how well the router and first hop are negotiating.

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Mon May 05, 2014 9:42 pm    Post subject: Reply with quote

Why not just leave the MTU at 576? Does that reduce bandwidth?
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Mon May 05, 2014 10:29 pm    Post subject: Reply with quote

grant123 wrote:
Why not just leave the MTU at 576? Does that reduce bandwidth?

grant123 ... a larger MTU is more efficent (larger data payload, smaller transmission packets) so an increase in throughput .... but its not as simple as it sounds (higher does not necessarily equal faster), TCP checks there is no transmission errors, and any packet that fails the check will be re-requested (at the same MTU) ... so if there are errors then there is an increase in the amount sent to aquire the same payload.

1500 is just the maximum allowed, and in most cases there is little reason to set it lower, in your case setting it to 567 avoids the problem, but you might also experience the same with 1400.

best ... khay
Back to top
View user's profile Send private message
grant123
l33t
l33t


Joined: 23 Mar 2005
Posts: 980

PostPosted: Tue May 06, 2014 9:50 pm    Post subject: Reply with quote

Got it, thank you for your help.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum