View previous topic :: View next topic |
Author |
Message |
li1_getoo l33t
![l33t l33t](/images/ranks/rank_rect_4.gif)
![](images/avatars/461639001403e7b6e1b04f.gif)
Joined: 20 Oct 2002 Posts: 661 Location: Queens , NY
|
Posted: Mon May 05, 2003 1:42 am Post subject: rsync etiquette guideline |
|
|
Code: | Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system. |
sometime u get blocked from using a mirror with a message that u used more then (x times) and u actually using it for the first time in that day or maybe even in a while , that kinda sxxxs
just a thought |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
csnyder n00b
![n00b n00b](/images/ranks/rank_rect_0.gif)
Joined: 18 Jun 2002 Posts: 44 Location: Grand Rapids, MI
|
Posted: Mon May 05, 2003 3:28 am Post subject: |
|
|
Another thought...
I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
shadow255 Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
![](images/avatars/11180770833e8dcd040ab57.jpg)
Joined: 04 Apr 2003 Posts: 412
|
Posted: Mon May 05, 2003 3:38 am Post subject: Re: rsync etiquette guideline |
|
|
li1_getoo wrote: | Code: | Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system. |
sometime u get blocked from using a mirror with a message that u used more then (x times) and u actually using it for the first time in that day or maybe even in a while , that kinda sxxxs
just a thought |
This is a perfect illustration of the proverb that says that a few rotten apples spoil the barrel. It goes without saying that setting up iptables rules to prevent users from repeatedly connecting to an rsync server is going to result in non-offenders getting screened off as well. I do not rsync more than twice in a week (barring the possibility of a series of vital security updates in a short span), but I will not be surprised to see myself getting an occasional refused connection as the recommendation gets adopted more.
The only solution that will please everyone is for the offenders to stop what they're doing. Oh wait, maybe it won't please them... oh well, I guess we all get to suffer because of the selfishness of a few. ![Rolling Eyes :roll:](images/smiles/icon_rolleyes.gif) _________________ Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
shadow255 Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
![](images/avatars/11180770833e8dcd040ab57.jpg)
Joined: 04 Apr 2003 Posts: 412
|
Posted: Mon May 05, 2003 3:40 am Post subject: |
|
|
csnyder wrote: | Another thought...
I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment. |
I don't suppose I speak for anyone but myself, but in my opinion, yes, you are breaching rsync etiquette. What do you expect to gain by rsyncing on an hourly basis? ![Shocked 8O](images/smiles/icon_eek.gif) _________________ Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
csnyder n00b
![n00b n00b](/images/ranks/rank_rect_0.gif)
Joined: 18 Jun 2002 Posts: 44 Location: Grand Rapids, MI
|
Posted: Mon May 05, 2003 3:44 am Post subject: |
|
|
The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
shadow255 Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
![](images/avatars/11180770833e8dcd040ab57.jpg)
Joined: 04 Apr 2003 Posts: 412
|
Posted: Mon May 05, 2003 3:53 am Post subject: |
|
|
csnyder wrote: | The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though. |
Exactly how difficult would it be for you to make a policy of having the server do a daily rsync, and then manually perform it when advisories are released after the daily one? Eight times a day to save a little effort still seems excessive to me. Not to mention that there have been dry spells on the order of nearly a week between GLSA announcements in the past. _________________ Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
aethyr Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/12934415273f99d0d61cd77.jpg)
Joined: 06 Apr 2003 Posts: 1085 Location: NYC
|
Posted: Mon May 05, 2003 3:59 am Post subject: |
|
|
I think it's safe to say there's no need to be rsync'ing more than twice a day and I can only hope that those people do it due to being unaware of the situation. Hopefully this will help raise awareness.
I see people on here who incessantly `emerge sync && emerge -u world` without realizing that they're wasting resources.
Gentoo's already bandwidth intensive enough without people hammering the servers...
For anyone interested in further reducing bandwidth on Gentoo servers, you might want to take a peek at this thread a couple weeks ago:
https://forums.gentoo.org/viewtopic.php?t=49139
I wonder how much bandwidth Gentoo consumes per capita versus some other distributions; that might be an interesting statistic. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
PowerFactor Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/4619806343f1242c842f38.jpg)
Joined: 30 Jan 2003 Posts: 1693 Location: out of it
|
Posted: Mon May 05, 2003 4:58 am Post subject: |
|
|
I agree that to be rsyncing more than twice a day on a regular basis is probably excessive. Realy, what advantage do you get by automatically rsyncing more than once a day. Like the message in the first post said, Quote: | There's being on the bleeding edge, and there is being just plain silly. | You can always rsync manually when there is a glsa, not like it's hard.
One other thing I see is, judging from conversations on these forums, people are a little too quick to nuke their distfiles in my opinion. Just because it is safe to do doesn't make it a good idea, it's certainly not courteous. By all means, clean out the cruft, but why load the distfiles servers excessively by downloading the same package more than once just because you were lazy. I like the distfile diffs idea. But I think we as users could do a lot better with the current system. And please be very quick to share your distfiles among machines if you have more than one. And copy them over if starting a new install. Little things like that, if we all do it , would help a lot I think. Am I completely off my rocker? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
aethyr Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/12934415273f99d0d61cd77.jpg)
Joined: 06 Apr 2003 Posts: 1085 Location: NYC
|
Posted: Mon May 05, 2003 6:16 am Post subject: |
|
|
PowerFactor:
I highly agree, which is why I use the python script found on the second page of this thread:
https://forums.gentoo.org/viewtopic.php?t=3011
This thread is all about cleaning out stale distfiles, which means that the distfiles that are still currently "in use" remain on your system unless they aren't used by any packages. This prevents re-downloading of the same files.
Again, like I've been saying to anyone who will listen, I think the key is to keep these distfiles on the harddrive and use small, incremental patches to patch them to newer versions.
Gentoo can be MUCH more bandwidth friendly (while still being source-based) than it is right now. I hope that the developers are keeping this in mind for the future. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
dol-sen Retired Dev
![Retired Dev Retired Dev](/images/ranks/rank-retired.gif)
![](images/avatars/895601024436ad996b979b.gif)
Joined: 30 Jun 2002 Posts: 2805 Location: Richmond, BC, Canada
|
Posted: Mon May 05, 2003 6:25 am Post subject: |
|
|
I have setup one of my boxes to serve via NSF its /usr/portage dir. It wasn't that hard and seems to work quite well. I just have to finnish some changes to my newer box and all will be using NFS to one /usr/portage. I rsync from 3 to 6 times per week depending what I'm doing or if I'm looking for some update to come thru. _________________ Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch... |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Lovechild Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/187805305641b5922f2f20b.gif)
Joined: 17 May 2002 Posts: 2858 Location: Århus, Denmark
|
Posted: Mon May 05, 2003 6:35 am Post subject: |
|
|
Hell, I break those guidelines daily at times.. but I'm not happy about it
Problems:
a) broken ebuilds
- Sometimes we have broken ebuilds in unstable, and those of us who act as willing testers for Gentoos unstable branch actually need to do emerge rsync a few times a day (also connected to d) at times.
b) incomplete rsyncing
- Rsync fails all the time for me, stalling halfway, thus I hit another server this can't be helped.
c) important updates
- if I see a GLSA notice I rsync... it's that simple.
d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago.
I can only conclude that rsync needs to die, and it has already be discussed in a previous GWN that we need to replace rsync with a less server demanding protocol - cvsup fx. but nothing has happened in the matter AFAIK. I'm known not to be rsync' greatest cheerleader.
Maybe we could get an option to pay for rsync services if we needed regular access, and then give the paying users prior access to mirrors (I'm guessing this can be done using iptables rules and some sort of authentication) - I would, for one, really hate to get the message, "You have reached your daily limit", and then just be kicked off the network. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Lovechild Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/187805305641b5922f2f20b.gif)
Joined: 17 May 2002 Posts: 2858 Location: Århus, Denmark
|
Posted: Mon May 05, 2003 6:38 am Post subject: |
|
|
aethyr wrote: | PowerFactor:
Again, like I've been saying to anyone who will listen, I think the key is to keep these distfiles on the harddrive and use small, incremental patches to patch them to newer versions.
|
I've been talking about this for ages, and the conclusion has always been that it's just to hard to implement. It would be a mighty cool feature, I agree. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
aethyr Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/12934415273f99d0d61cd77.jpg)
Joined: 06 Apr 2003 Posts: 1085 Location: NYC
|
Posted: Mon May 05, 2003 7:06 am Post subject: |
|
|
Lovechild, two points:
#1, updating through patching is already being done in other source distributions like sorcerer/sourcery (dunno if sourcemage is doing it), so it can't be impossible (again, I discussed possible implementation strategies, feel free to brainstorm in that thread, I might try to hack a prototype model when I'm finished this semester).
#2, if everyone kept their rsync usage to a reasonable level there'd probably be no need to get denied, or have to pay. I think raising awareness through the GWN, forums, and even on the mirrors is a valid way to get this message across. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Lovechild Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/187805305641b5922f2f20b.gif)
Joined: 17 May 2002 Posts: 2858 Location: Århus, Denmark
|
Posted: Mon May 05, 2003 7:18 am Post subject: |
|
|
1) I know - but that don't change the fact that it's hard to do.. I've been looking at how to go about it, and I have a few promising ideas, but the best option might be to simply enforce the use of a "base package" and then release everything from there as diffs, the kernel fx. already does this.
2) I also know that - I would have no problem paying because I know I use the mirrors more than most people.. I was talking to a few people a while back about replacing and Gentoo transfers with BitTorrent to surcome this saturation issue and I think it would be an exellent approach for the file transfers but the rsyncing is a bit trickier to do, a sidebenefit here would be that with bittorrent it would be DEAD easy to setup a mirror, all you would have to do was to start bittorrent and point it to the tracker, as soon as you were done downloading you would automatically become a mirror, and would stay that way untill bittorrent was closed. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
aethyr Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/12934415273f99d0d61cd77.jpg)
Joined: 06 Apr 2003 Posts: 1085 Location: NYC
|
Posted: Mon May 05, 2003 8:19 am Post subject: |
|
|
on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).
All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
amne Bodhisattva
![Bodhisattva Bodhisattva](/images/ranks/rank-bodhisattva.gif)
![](images/avatars/117771253447ad6e0ae3631.gif)
Joined: 17 Nov 2002 Posts: 6378 Location: Graz / EU
|
Posted: Mon May 05, 2003 8:36 am Post subject: |
|
|
one thing about reporting problems with broken mirrors:
i once connected to a rsync mirror that downgraded my portage tree to a >1 week old version, but i did not report this as a bug.
reason: i did not know which mirror it was as this information hat scrolled out of the terminal (and out of its buffer) in the meanwhile. since then i usually try to take a look at the mirror. maybe emerge sync should print the server information at the end and/or log it to /var/log/emerge.log, too. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
klieber Bodhisattva
![Bodhisattva Bodhisattva](/images/ranks/rank-bodhisattva.gif)
![](images/avatars/71602131143de10f19aaec.jpg)
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Mon May 05, 2003 10:25 am Post subject: |
|
|
csnyder wrote: | Another thought...
I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment. |
Yes. It is.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
klieber Bodhisattva
![Bodhisattva Bodhisattva](/images/ranks/rank-bodhisattva.gif)
![](images/avatars/71602131143de10f19aaec.jpg)
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Mon May 05, 2003 10:26 am Post subject: |
|
|
aethyr wrote: | Gentoo can be MUCH more bandwidth friendly (while still being source-based) than it is right now. I hope that the developers are keeping this in mind for the future. |
Yes. We are.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
TheCoop Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
![](images/avatars/50962178541c2cd784cf26.jpg)
Joined: 15 Jun 2002 Posts: 1814 Location: Where you least expect it
|
Posted: Mon May 05, 2003 10:33 am Post subject: |
|
|
There was some discussion about a p2p file sharing system (volunteers only ofc), so a gentoo install could share one or two files, so most of the main distfile servers could be turned into rsync servers which would help the load somewhat...
Maybe it could be adapted for a p2p rsync, a rsync server on a standard gentoo install that would only allow one connection at a time... _________________ 95% of all computer errors occur between chair and keyboard (TM)
"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler
Change the world - move a rock |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
klieber Bodhisattva
![Bodhisattva Bodhisattva](/images/ranks/rank-bodhisattva.gif)
![](images/avatars/71602131143de10f19aaec.jpg)
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Mon May 05, 2003 10:39 am Post subject: |
|
|
To follow up on the GWN article and clarify a few points:- These are guidelines, not rules. We are asking for your help in keeping our rsync mirror system sustainable for the future. That said, if you occasionally have to sync a third time in a 24 hour period, you're not going to have jack-booted thugs show up on your doorstep with a cluebat...
- Only the most aggregious users will be rate limited. Syncing more than twice a day won't automatically get you rate limited. Syncing against one particular server, 50 times per day almost certainly will get you rate limited. Don't be stupid and you don't have to worry about it.
- Don't single out particular servers I can't stress this one enough. Not only is it poor etiquette, it's stupid as well -- you draw attention to yourself because your IP address hits the logs of that server over and over. If you use rsync.<continent>.gentoo.org or rsync.<country>.gentoo.org, you'll spread your usage over many servers. This is a Good Thing.
This is a very real problem, folks. One of our German mirror admins ran a script against his logs and found that the top 5 users were consuming over 50% of the CPU resources going to rsync on that machine. (Those users have all been rate limited, btw.) We're not trying to be jerks about this -- but when I have multiple rsync mirror admins complaining to me about the amount of resources we're taking up, we have to do something about it. With your cooperation and assistance, this GWN article will be all we have to do.
One final point to Lovechild's question/suggestion on paid access to rsync mirrors. That's something we're also looking at doing. The issue comes down to building out the payment processing infrastructure. It's on the radar and something that I'd like to see us do. I just don't know when we'll have the time or resources to do so.
--kurt _________________ The problem with political jokes is that they get elected
Last edited by klieber on Mon May 05, 2003 12:36 pm; edited 1 time in total |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
klieber Bodhisattva
![Bodhisattva Bodhisattva](/images/ranks/rank-bodhisattva.gif)
![](images/avatars/71602131143de10f19aaec.jpg)
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Mon May 05, 2003 10:44 am Post subject: |
|
|
Lovechild wrote: | d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago. |
Have you considered waiting 4 or even (gasp) 6 hours to let the changes propogate naturally? All the rsync mirrors sync every 30 minutes against one master rsync mirror (rsync1.us.gentoo.org) so all mirrors should have new updates within 30 minutes of them hitting the tree.
That said, I fail to see how waiting a few hours to let the changes work their way down is really that onerous of a request. IMO, the rest of your points were valid -- this one is bogus.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
fishhead Apprentice
![Apprentice Apprentice](/images/ranks/rank_rect_2.gif)
![](images/avatars/1205207263edfff51a6731.jpg)
Joined: 07 Mar 2003 Posts: 162 Location: Pasadena, CA
|
Posted: Mon May 05, 2003 11:04 am Post subject: |
|
|
Perhaps it would help to have emerge check (by default) how many rsyncs have been done in a 24 hour period and print out a message after the fourth or fifth time. Part of the problem may be that users just aren't THINKING about what they are doing. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Lovechild Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/187805305641b5922f2f20b.gif)
Joined: 17 May 2002 Posts: 2858 Location: Århus, Denmark
|
Posted: Mon May 05, 2003 12:31 pm Post subject: |
|
|
aethyr wrote: | on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).
All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements. |
Ah, but all the mirrors are still running bittorrent, so the bw. is actually better even when it's an "unpopular" file... like emacs or something
Bittorrent would make it a lot easier to cope with sudden rushs - like when the new versions of KDE, xfree and GNOME are released - big popular packages. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Lovechild Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/187805305641b5922f2f20b.gif)
Joined: 17 May 2002 Posts: 2858 Location: Århus, Denmark
|
Posted: Mon May 05, 2003 12:40 pm Post subject: |
|
|
One question on the load spreading using set mirrors.
I know I can set GENTOO_MIRRORS in make.conf - but I thought rsync already pointed to a master hub that spread the load to a random non busy mirror.
Anyways if this is not the case, then what's the syntax we need to add, and where's the "official" list of rsync mirrors (I assume we have one someplace) so we can change it to aid the cause.
-edit-
I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
klieber Bodhisattva
![Bodhisattva Bodhisattva](/images/ranks/rank-bodhisattva.gif)
![](images/avatars/71602131143de10f19aaec.jpg)
Joined: 17 Apr 2002 Posts: 3657 Location: San Francisco, CA
|
Posted: Mon May 05, 2003 12:52 pm Post subject: |
|
|
Lovechild wrote: | I know I can set GENTOO_MIRRORS in make.conf - but I thought rsync already pointed to a master hub that spread the load to a random non busy mirror. |
It's not a master hub -- it's handled through DNS via a round robin system.
Lovechild wrote: | I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm |
See this Tips and Tricks. You can change your SYNC variable in /etc/make.conf to something that is geographically closer to you. Personally, I encourage all users to use the continent-level rsync rotations.
--kurt _________________ The problem with political jokes is that they get elected |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|