| View previous topic :: View next topic |
| Author |
Message |
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
 Posted: Mon Feb 08, 2010 8:02 pm Post subject: emerge --rsync problem |
 |
|
After not so fortunate conversation on irc.gentoo.org i get partial solution of my problem.
I couldn't get emerge --rsync to work at all until i added additional parameter --bwlimit, which is currently set at very low value like 10 kbp/sec.
All other things remained unchanged from what is written in profile desktop profile 2010.
Without --bwlimit i get speed of around 100-200 bytes/ sec and connection is lost after half and hour or so. Also ocassionally i may get corruption of my filesystem ext3.
Addendum
Well, I just saw there is similar problem on the sticky post, just I couldn't really understand what is the solution.
The line in c file which i get error is different. But all other symptoms are the same : rsync error: error in rsync protocol data stream
Mirrorselect is not the solution. I can find server no matter of which mirror i'm using. The problem is in the stream itself which get stucked.
Last edited by goretz on Mon Feb 08, 2010 8:16 pm; edited 1 time in total |
|
| Back to top |
|
 |
msalerno
Veteran
Joined: 17 Dec 2002
Posts: 1336
Location: Sweating in South Florida
|
| Posted: Mon Feb 08, 2010 8:13 pm Post subject: |
|
|
Have you tried looking for a different rsync mirror? Try a few mirrors and see if you get the same results.
http://en.gentoo-wiki.com/wiki/Mirrorselect
Regarding ext3 corruption, I would recommend fsck on your unmounted filesystems and making sure your disk isn't about to take a dump.
_________________
When harmonious relationships dissolve
Then respect and devotion arise;
When a nation falls to chaos
Then loyalty and patriotism are born.
-Lao Tse |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Mon Feb 08, 2010 8:22 pm Post subject: |
|
|
Addendum
Well, I just saw there is similar problem on the sticky post, just I couldn't really understand what is the solution.
The line in c file which i get error is different. But all other symptoms are the same : rsync error: error in rsync protocol data stream
Mirrorselect is not the solution. I can find server no matter of which mirror i'm using. The problem is in the stream itself which get stucked.
Also i tried with option --dry-run with the same --bwlimit and it seemed considerably faster.
My real question what should be "normal" values of --bwlimit if my bandwidth "other internet" line is max 192 kb/sec ? |
|
| Back to top |
|
|
gerard82
Veteran
Joined: 04 Jan 2004
Posts: 1927
Location: Netherlands
|
| Posted: Mon Feb 08, 2010 8:27 pm Post subject: |
|
|
I've never before seen anything like
And I can't find it in man emerge either.
Shouldn't that be
Gerard.
_________________
To install Gentoo I use sysrescuecd.Based on Gentoo,has Firefox to browse Gentoo docs and mc to browse (and edit) files.
The same disk can be used for 32 and 64 bit installs.
You can follow the Handbook verbatim.
http://www.sysresccd.org/Download |
|
| Back to top |
|
|
msalerno
Veteran
Joined: 17 Dec 2002
Posts: 1336
Location: Sweating in South Florida
|
| Posted: Mon Feb 08, 2010 8:28 pm Post subject: |
|
|
No idea, I never had to specify a limit. What kind of throughput do you get with an emerge-webrsync? Are you behind a firewall or something doing QOS?
_________________
When harmonious relationships dissolve
Then respect and devotion arise;
When a nation falls to chaos
Then loyalty and patriotism are born.
-Lao Tse |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Mon Feb 08, 2010 8:30 pm Post subject: |
|
|
Yes, it should be emerge --sync. My bad
I made this mistake in the post but not when running my observations 
This is probably cause i know already that emerge --sync calling rsync. |
|
| Back to top |
|
|
msalerno
Veteran
Joined: 17 Dec 2002
Posts: 1336
Location: Sweating in South Florida
|
| Posted: Mon Feb 08, 2010 8:31 pm Post subject: |
|
|
Good catch gerard82. I have not issued an emerge --rsync in many years.
_________________
When harmonious relationships dissolve
Then respect and devotion arise;
When a nation falls to chaos
Then loyalty and patriotism are born.
-Lao Tse |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Mon Feb 08, 2010 8:55 pm Post subject: |
|
|
I may also add that the problem boils down to rsync version 3.0.6 ( may be other versions too) probably kernel and fs interactions between them.
Not really emerge or portage issues. |
|
| Back to top |
|
|
John R. Graham
Administrator
Joined: 08 Mar 2005
Posts: 4849
Location: Somewhere over Atlanta, Georgia
|
| Posted: Mon Feb 08, 2010 9:46 pm Post subject: |
|
|
Might want to try adding | Code: | | PORTAGE_RSYNC_EXTRA_OPTS="-vvv" |
to your make.conf. This will show extra debugging information while rsync is running and may give us the clues needed to help you. Two more comments:- It is curious that you're getting filesystem corruption. rsync shouldn't be able to do this by itself. It seems certain to me that there's something else going on that you haven't diagnosed. Before I proceeded, I'd load your rescue CD of choice and run an fsck (and, perhaps, badblocks) on your filesystem, if I were you.
- I've seen TCP/IP break down on extremely slow, noisy networks where a lot of retries are occurring in the protocol stack. This is because TCP is a Sliding Window Protocol. That limiting the bandwidth contributes to the robustness of the rsync run is indirect evidence that packets that are still pending are sliding outside of the window (and thus being dropped). It makes me wonder if you're having network issues with any other program. Are you?
- John
_________________
Yoda: "Intentionally left blank, this space is." |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Mon Feb 08, 2010 11:46 pm Post subject: |
|
|
John I did added that extra verbose feature, not that i was able to diagnose what is wrong though
I may retreat from my previous words about --bwlimit being important at all.
Now I'm again unable to make emerge --rsync no matter what is or is not --bwlimit
As for trashing FS , I suppose it is possible somehow if emerge --rsync doesn't work or is working erratically. Since it is making a lot of timestamps on FS structure and when I'm getting erratic traffic I suppose it can go wrong somehow in FS as well. |
|
| Back to top |
|
|
John R. Graham
Administrator
Joined: 08 Mar 2005
Posts: 4849
Location: Somewhere over Atlanta, Georgia
|
| Posted: Tue Feb 09, 2010 12:15 am Post subject: |
|
|
Well, in all humility, I'm suggesting that, if you post some actual verbose error messages, then I might be able to diagnose what's wrong. Why don't you give it a try?
- John
_________________
Yoda: "Intentionally left blank, this space is." |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Tue Feb 09, 2010 2:05 am Post subject: |
|
|
I found open bug http://bugs.gentoo.org/show_bug.cgi?id=295572 which might be related to my problem.
John I don't have errors exept that connection lost and it can report different line results in C code i.o lines.
It is not trivial to find out what is really happening ( unless you are seasoned gentoo developer of course ) |
|
| Back to top |
|
|
Hu
Watchman
Joined: 06 Mar 2007
Posts: 6828
|
| Posted: Tue Feb 09, 2010 4:00 am Post subject: |
|
|
| goretz wrote: | | As for trashing FS , I suppose it is possible somehow if emerge --rsync doesn't work or is working erratically. Since it is making a lot of timestamps on FS structure and when I'm getting erratic traffic I suppose it can go wrong somehow in FS as well. |
It is generally held that in the absence of software bugs or hardware failure, no unprivileged program can induce the operating system to misbehave, whether in the form of a panic or as filesystem corruption. While I might agree if you were using an unusual or known unstable filesystem, ext3 is old and well respected. I would be surprised if ext3 contained a corruption bug triggered simply by an rsync workload. Such a bug would need to be time sensitive, or it would have affected many other Gentoo users as well. I think it is far more likely that your hardware is slowly failing. |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Tue Feb 09, 2010 9:40 am Post subject: |
|
|
Hu
While I'm generally perfectly agree with your statements let's try to dissect them. You wrote
| Quote: | | It is generally held that in the absence of software bugs or hardware failure, no unprivileged program |
1. We don't know whether it is software or hardware fault, still. That is what I'm going to find out
Although i didn't run MEMTEST86 for 24 hours or bonnie+ extensive stresses as i was suggested to do on irc.gentoo.org b.t.w.
But I do know that this same hardware worked well just a couple of days ago under different OS ( both WIN7 and Debian stable )
2. rsync is really "very privileged" piece of software , so saying
| Quote: | | triggered simply by an rsync workload |
is a bit of understatement.
I dig a bit more and found some new info to further base my claims
By default PORTAGE_RSYNC_OPTS includes this option --recursive
Now, if you carefully read man pages for rsync ( Very entertaining ) you may find interesting things
like you need both client and server to be running at least version 3.0 ( I suppose that may be the reason why some mirrors stop working) moreover if read carefully there you can also deduce that our in-house gentoo installation actually using --delete-during when specifying --delete.
If you go to rsync main webpage you may also find another bit of info like this :
| Quote: | NEWS for rsync 3.0.7 (31 Dec 2009)
Protocol: 30 (unchanged)
Changes since 3.0.6:
BUG FIXES:
- Fixed a bogus free when using --xattrs with --backup.
- Avoid an error when --dry-run was trying to stat a prior hard-link file
that hasn't really been created.
- Fixed a problem with --compress (-z) where the receiving side could
return the error "inflate (token) returned -5".
- Fixed a bug where --delete-during could delete in a directory before it
noticed that the sending side sent an I/O error for that directory (both
sides of the transfer must be at least 3.0.7).
|
And other bug fixes
So we have a potential for triggering some bug ! I'm not saying that exactly this bug, but you can see a potential already.
We also need I/O error. I would call it silent mutation, since myself coming from biology field.
Apparently my setup has more silent mutations like this and the result is - corrupted FS
ext3 indeed is trusted and long used filesystem. But for filesystem to function there is a lot of code in the kernel. And we are using not that much tested code like say RED HAT Enterprice installs.
To finish this up : I'm again thinking that --bwlimit can help.
And i forgot to mention from the begining that --bwlimit advice i took from somebody on irc, and he also experienced fs corruptions.
But I may one day say - yes indeed it was my faulty hardware after all ... |
|
| Back to top |
|
|
Jaglover
Advocate
Joined: 29 May 2005
Posts: 3404
Location: Saint Amant, Acadiana
|
| Posted: Tue Feb 09, 2010 9:53 am Post subject: |
|
|
| I'd pay close attention to the hard drive, is it developing bad sectors. Failing HDD (if true) would explain all your troubles, incl damaged malfunctioning binaries. |
|
| Back to top |
|
|
msalerno
Veteran
Joined: 17 Dec 2002
Posts: 1336
Location: Sweating in South Florida
|
| Posted: Tue Feb 09, 2010 2:07 pm Post subject: |
|
|
rather than using --bwlimit, why not just emerge-webrsync ?
The ext3 code from the redhat kernel is most likely the same exact code in the vanilla sources, not to mention it has been and is currently used by tens of/hundres of thousands of linux installs regardless of distro. Rsync and ext3 are used all over the place too.
Not sure what kind of drive you are using, but have you tried smartmontools? or even the manufacturer diag tools?
_________________
When harmonious relationships dissolve
Then respect and devotion arise;
When a nation falls to chaos
Then loyalty and patriotism are born.
-Lao Tse |
|
| Back to top |
|
|
Hu
Watchman
Joined: 06 Mar 2007
Posts: 6828
|
| Posted: Wed Feb 10, 2010 4:11 am Post subject: |
|
|
| goretz wrote: | 2. rsync is really "very privileged" piece of software , so saying
| Quote: | | triggered simply by an rsync workload |
is a bit of understatement. |
I disagree. You can, and should, use FEATURES=usersync, in which case rsync will not be running with privilege. However, even if it were running with root privilege, it would still need to do some privileged call in order to destabilize the system if we assume an absence of hardware failures and software bugs. As far as I know, nothing in the Portage rsync workload should initiate any such privileged calls. |
|
| Back to top |
|
|
goretz
n00b
Joined: 24 Dec 2008
Posts: 26
|
| Posted: Wed Feb 10, 2010 6:41 pm Post subject: |
|
|
Hu:
This might be off topic but the feature you mentioned is disabled by default.
| Quote: | | You can, and should, use FEATURES=usersync |
If I understand this correctly
| Quote: | *portage-2.1.6.6 (14 Jan 2009)
14 Jan 2009; Zac Medico <zmedico@gentoo.org> +portage-2.1.6.6.ebuild:
2.1.6.6 bump. This fixes bug #254825 (repoman output for changed
distfiles digests) and bug #254860 (AttributeError triggered by
`emerge -u <atom>`. Also, dropping of privileges for emerge --sync is
now controlled by FEATURES=usersync, which is disabled by default. |
At least it is definitely not in 2010 desktop profile. |
|
| Back to top |
|
|
|
Portage & Programming
