View previous topic :: View next topic |
Author |
Message |
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Tue Aug 21, 2012 2:49 pm Post subject: Emerge --Sync stopped working. |
|
|
All, I have seen a number of posts that mention if rsync does not work for you (emerge --sync) that "emerge-webrsync" can get you a portage snapshot.
My Gentoo servers at home used to "emerge --sync" once a night, and at some time ago (a few months) it stopped working.
Now I use a Cisco 1841 with reflexive ACL's as my firewall, with a dedicated static IP for the main server. I have visibility over the network bits, what I would like to do is understand why the "emerge --sync" using rsync stopped working.
I saw a post somewhere that the newer rsync client changed it's behavior, but posts were not specific as to what the behavior was?
Anyone take a deeper look as to what changed with rsync?
Nick |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9713 Location: almost Mile High in the USA
|
Posted: Tue Aug 21, 2012 3:29 pm Post subject: |
|
|
What does it do when you run it manually instead of in a cron job?
I have my server emerge --sync every few days, not every night - but it still seems to be working... All my other machines sync off of my server... I have a static IP with "consumer/home" networking equipment... I don't firewall off rsync or web ports... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Tue Aug 21, 2012 9:18 pm Post subject: |
|
|
eccerr0r wrote: | What does it do when you run it manually instead of in a cron job?
I have my server emerge --sync every few days, not every night - but it still seems to be working... All my other machines sync off of my server... I have a static IP with "consumer/home" networking equipment... I don't firewall off rsync or web ports... |
Just the excessive retries... then bails.
/etc/make.conf
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
PORTAGE_RSYNC_OPTS="--progress --recursive --links --safe-links --compress --perms --times --force --whole-file --itemize-changes --delete --delete-during --stats --timeout=1800 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/overlay"
GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.gossamerhost.com http://gentoo.llarian.net/"
gubbie ~ # emerge --sync
>>> Starting rsync with rsync://208.100.4.53/gentoo-portage...
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying...
>>> Starting retry 1 of 18 with rsync://64.59.140.91/gentoo-portage
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying...
>>> Starting retry 2 of 18 with rsync://209.59.138.21/gentoo-portage
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying...
>>> Starting retry 3 of 18 with rsync://209.221.142.124/gentoo-portage
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying... |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9713 Location: almost Mile High in the USA
|
Posted: Tue Aug 21, 2012 9:45 pm Post subject: |
|
|
I think rsync uses port 873, is this blocked in your firewall?
I don't recall there being any changes...then again I have a fairly unrestrictive outgoing firewall. Incoming is a bit different... |
|
Back to top |
|
|
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Tue Aug 21, 2012 9:52 pm Post subject: |
|
|
eccerr0r wrote: | I think rsync uses port 873, is this blocked in your firewall? |
No, it is not.
My firewall is a Cisco 1841, so I am using reflexive ACL's that premit the outbound request, and create a reverse rule that get's added to the inbound ACL.
DSL-Router#sho access-list traffic | inc 134.161.116.17
permit tcp host 134.161.116.17 eq 873 host 69.30.73.18 eq 36935 (11 matches) (time left 295)
In this case, the poorman's state engine (reflexive list entry) was created, and even saw 11 packets come back from this particular gentoo rsync server..
Nick |
|
Back to top |
|
|
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Tue Aug 21, 2012 9:54 pm Post subject: |
|
|
nellson wrote: | eccerr0r wrote: | I think rsync uses port 873, is this blocked in your firewall? |
No, it is not.
My firewall is a Cisco 1841, so I am using reflexive ACL's that premit the outbound request, and create a reverse rule that get's added to the inbound ACL.
DSL-Router#sho access-list traffic | inc 134.161.116.17
permit tcp host 134.161.116.17 eq 873 host 69.30.73.18 eq 36935 (11 matches) (time left 295)
In this case, the poorman's state engine (reflexive list entry) was created, and even saw 11 packets come back from this particular gentoo rsync server..
Nick |
And if I look at the RSYNC port:
DSL-Router#sho access-list traffic | inc eq 873
permit tcp host 64.59.140.91 eq 873 host 69.30.73.18 eq 40563 (8 matches) (time left 299)
permit tcp host 208.70.246.16 eq 873 host 69.30.73.18 eq 36811 (12 matches) (time left 291)
permit tcp host 216.194.64.133 eq 873 host 69.30.73.18 eq 58534 (17 matches) (time left 298)
permit tcp host 216.165.129.134 eq 873 host 69.30.73.18 eq 47525 (13 matches) (time left 287)
permit tcp host 209.59.138.21 eq 873 host 69.30.73.18 eq 60070 (13 matches) (time left 280)
permit tcp host 128.175.60.112 eq 873 host 69.30.73.18 eq 48501 (12 matches) (time left 283)
permit tcp host 129.21.171.98 eq 873 host 69.30.73.18 eq 43443 (14 matches) (time left 299)
permit tcp host 208.100.4.53 eq 873 host 69.30.73.18 eq 36078 (14 matches) (time left 268)
permit tcp host 209.221.142.124 eq 873 host 69.30.73.18 eq 52969 (14 matches) (time left 267)
permit tcp host 128.61.111.9 eq 873 host 69.30.73.18 eq 44216 (14 matches) (time left 261)
permit tcp host 134.161.116.17 eq 873 host 69.30.73.18 eq 36935 (14 matches) (time left 227)
permit tcp host 128.10.252.13 eq 873 host 69.30.73.18 eq 43154 (3 matches) (time left 128)
permit tcp host 156.56.247.193 eq 873 host 69.30.73.18 eq 33529 (14 matches) (time left 233)
permit tcp host 67.212.64.3 eq 873 host 69.30.73.18 eq 39016 (14 matches) (time left 196)
You see that they all got replies...
And this used to work. I have not mucked with my router in quite a while..
Nick |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9713 Location: almost Mile High in the USA
|
Posted: Tue Aug 21, 2012 10:14 pm Post subject: |
|
|
Did you setup firewall stuff on your local machine recently?
After reading your post I went to look at my firewall and found it was a bit more relaxed than I thought, and my local gentoo rsync mirror is actually visible from the outside. While it's not a real big problem, I didn't expect it...
Might want to run tcpdump or wireshark on your ether port and see if anything's getting back to your machine...
Does other stuff work (web, etc.)? is it only rsync that's broken? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Thu Aug 23, 2012 12:01 am Post subject: |
|
|
eccerr0r wrote: | Did you setup firewall stuff on your local machine recently?
After reading your post I went to look at my firewall and found it was a bit more relaxed than I thought, and my local gentoo rsync mirror is actually visible from the outside. While it's not a real big problem, I didn't expect it...
Might want to run tcpdump or wireshark on your ether port and see if anything's getting back to your machine...
Does other stuff work (web, etc.)? is it only rsync that's broken? |
Well.. So far just rsync.
<sigh> Wow... so I set up TCPDUMP, and in another session started the "emerge --sync" and the first site it hit just worked.. :-\
I really wanted to dog this down and post most details of a way to fix this. But if it won't stay broke...
I will watch it for a few days... if it breaks I will post the tcpdump.
I have never set up my local gentoo firewall (ipchains??) so if it got updated in a previous emerge world, I would have missed that.. good idea to check though!
Nick
Nick |
|
Back to top |
|
|
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Fri Aug 24, 2012 1:41 pm Post subject: |
|
|
OK, so next day the problem is back. Only this time i have TCPDUMP working for me.
So from my server point of view, it never sees a reply.
Here is the server sending out the request:
Code: |
gubbie ~ # tcpdump -i eth0 tcp port 873
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
06:32:50.351026 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [S], seq 2730467521, win 5840, options [mss 1460,sackOK,TS val 1034299160 ecr 0,nop,wscale 6], length 0
06:32:50.407702 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [S.], seq 4117327491, ack 2730467522, win 14480, options [mss 1460,sackOK,TS val 39555318 ecr 1034299160,nop,wscale 9], length 0
06:32:50.407899 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 1, win 92, options [nop,nop,TS val 1034299174 ecr 39555318], length 0
06:32:50.408266 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 1:15, ack 1, win 92, options [nop,nop,TS val 1034299174 ecr 39555318], length 14
06:32:50.457666 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [.], ack 15, win 29, options [nop,nop,TS val 39555331 ecr 1034299174], length 0
06:32:50.912943 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39555445 ecr 1034299174], length 14
06:32:50.913053 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034299300 ecr 39555445], length 0
06:32:50.913510 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034299300 ecr 39555445], length 15
06:32:51.161180 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39555507 ecr 1034299174], length 14
06:32:51.161276 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034299362 ecr 39555507,nop,nop,sack 1 {1:15}], length 0
06:32:51.165686 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034299363 ecr 39555507], length 15
06:32:51.656830 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39555631 ecr 1034299174], length 14
06:32:51.656926 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034299486 ecr 39555631,nop,nop,sack 1 {1:15}], length 0
06:32:51.670605 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034299489 ecr 39555631], length 15
06:32:52.648684 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39555879 ecr 1034299174], length 14
06:32:52.648780 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034299734 ecr 39555879,nop,nop,sack 1 {1:15}], length 0
06:32:52.676009 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034299741 ecr 39555879], length 15
06:32:54.637273 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39556376 ecr 1034299174], length 14
06:32:54.637384 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034300231 ecr 39556376,nop,nop,sack 1 {1:15}], length 0
06:32:54.694934 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034300245 ecr 39556376], length 15
06:32:58.612848 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39557370 ecr 1034299174], length 14
06:32:58.612910 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034301225 ecr 39557370,nop,nop,sack 1 {1:15}], length 0
06:32:58.726412 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034301253 ecr 39557370], length 15
06:33:02.209303 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [F.], seq 30, ack 15, win 92, options [nop,nop,TS val 1034302124 ecr 39557370], length 0
06:33:06.572702 IP 140.211.166.189.rsync > 10.0.0.22.35819: Flags [P.], seq 1:15, ack 15, win 29, options [nop,nop,TS val 39559360 ecr 1034299174], length 14
06:33:06.572836 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [.], ack 15, win 92, options [nop,nop,TS val 1034303215 ecr 39559360,nop,nop,sack 1 {1:15}], length 0
06:33:06.792342 IP 10.0.0.22.35819 > 140.211.166.189.rsync: Flags [P.], seq 15:30, ack 15, win 92, options [nop,nop,TS val 1034303269 ecr 39559360], length 15
|
Here is my 1841 router with "debug ip packet 123" where ACL 123 is
access-list 123 permit tcp any any eq 873
access-list 123 permit tcp any eq 873 any
ip nat inside source static 10.0.0.22 69.30.73.18 <- for reference.
It tracks the flow going out....
Code: |
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, Stateful Inspection(4), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, CCE Input Classification(5), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, QoS Drop(6), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, Ingress-NetFlow(17), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, Virtual Fragment Reassembly(21), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, Virtual Fragment Reassembly After IPSec Decryption(32), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, Policy Routing(58), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189, len 60, input feature, MCI Check(63), rtype 0, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, CCE Output Classification(5), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, Post-routing NAT Outside(17), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, Stateful Inspection(20), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, Firewall (NAT)(33), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, IPsec or interface ACL checked on pre-encrypted cleartext packets(34), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, Firewall (inspect)(38), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, Post-Ingress-NetFlow(52), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, output feature, Egress-Netflow(56), rtype 1, forus FALSE, sendself FALSE, mtu 0
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), g=69.30.73.1, len 60, forward
Aug 24 06:32:50: IP: s=69.30.73.18 (FastEthernet0/0.1), d=140.211.166.189 (BVI1), len 60, sending full packet
|
And the reflexive ACL get's built and sees the return hits...
Code: |
DSL-Router#sho access-list traffic | inc 873
permit tcp host 140.211.166.189 eq 873 host 69.30.73.18 eq 35819 (10 matches) (time left 299)
|
and when I get clever and try just port testing:
Code: |
gubbie ~ # telnet 140.211.166.189 873
Trying 140.211.166.189...
Connected to 140.211.166.189.
Escape character is '^]'.
@RSYNCD: 30.0
Welcome to bobolink.gentoo.org / rsync.gentoo.org
Server Address : 140.211.166.189
Contact Name : mirror-admin@gentoo.org
Hardware : 4 x Intel(R) Atom(TM) CPU D510 @ 1.66GHz, 3960MB RAM
Sponsor : Gentoo Linux, Open Source Lab, Corvallis, OR, USA
Please note: common gentoo-netiquette says you should not sync more
than once a day. Users who abuse the rsync.gentoo.org rotation
may be added to a temporary ban list.
MOTD autogenerated by update-rsync-motd on Sun Apr 1 01:06:12 UTC 2012
|
A TCP connect from my server also generates the same traces above, and appears to be good!
So there is something going wonky in the rsync protocol that it failing... anyone got a way to debug RSYNC?
Nick |
|
Back to top |
|
|
nellson n00b
Joined: 24 Jun 2004 Posts: 66
|
Posted: Fri Aug 31, 2012 2:47 pm Post subject: |
|
|
Not sure why, but for the time being rsync is working reliably again. Wish I have something more concrete to report. :-\ |
|
Back to top |
|
|
jrussia Tux's lil' helper
Joined: 29 Aug 2012 Posts: 89 Location: Chicago
|
Posted: Fri Aug 31, 2012 3:56 pm Post subject: |
|
|
I noticed you have SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage" in make.conf...
I don't get any results from pinging rsync.namerica.gentoo.org, but rsync.us.gentoo.org and rsync.ca.gentoo.org are pinging fine. Maybe choose one of those and replace. Maybe the namerica name was changed a few months ago (I don't see it in mirrorselect.)
-js
edit: now it pinged fine. Maybe it's just overloaded and that's causing your intermittent problems. I am using the rsync.us.gentoo.org link w/o problems, so maybe it's worth trying. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|