Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rsync etiquette guideline
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
li1_getoo
l33t
l33t


Joined: 20 Oct 2002
Posts: 661
Location: Queens , NY

PostPosted: Mon May 05, 2003 1:42 am    Post subject: rsync etiquette guideline Reply with quote

Code:
Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system.


sometime u get blocked from using a mirror with a message that u used more then (x times) and u actually using it for the first time in that day or maybe even in a while , that kinda sxxxs
just a thought
Back to top
View user's profile Send private message
csnyder
n00b
n00b


Joined: 18 Jun 2002
Posts: 44
Location: Grand Rapids, MI

PostPosted: Mon May 05, 2003 3:28 am    Post subject: Reply with quote

Another thought...

I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment.
Back to top
View user's profile Send private message
shadow255
Guru
Guru


Joined: 04 Apr 2003
Posts: 412

PostPosted: Mon May 05, 2003 3:38 am    Post subject: Re: rsync etiquette guideline Reply with quote

li1_getoo wrote:
Code:
Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system.


sometime u get blocked from using a mirror with a message that u used more then (x times) and u actually using it for the first time in that day or maybe even in a while , that kinda sxxxs
just a thought

This is a perfect illustration of the proverb that says that a few rotten apples spoil the barrel. It goes without saying that setting up iptables rules to prevent users from repeatedly connecting to an rsync server is going to result in non-offenders getting screened off as well. I do not rsync more than twice in a week (barring the possibility of a series of vital security updates in a short span), but I will not be surprised to see myself getting an occasional refused connection as the recommendation gets adopted more.

The only solution that will please everyone is for the offenders to stop what they're doing. Oh wait, maybe it won't please them... oh well, I guess we all get to suffer because of the selfishness of a few. :roll:
_________________
Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy
Back to top
View user's profile Send private message
shadow255
Guru
Guru


Joined: 04 Apr 2003
Posts: 412

PostPosted: Mon May 05, 2003 3:40 am    Post subject: Reply with quote

csnyder wrote:
Another thought...

I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment.

I don't suppose I speak for anyone but myself, but in my opinion, yes, you are breaching rsync etiquette. What do you expect to gain by rsyncing on an hourly basis? 8O
_________________
Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy
Back to top
View user's profile Send private message
csnyder
n00b
n00b


Joined: 18 Jun 2002
Posts: 44
Location: Grand Rapids, MI

PostPosted: Mon May 05, 2003 3:44 am    Post subject: Reply with quote

The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though.
Back to top
View user's profile Send private message
shadow255
Guru
Guru


Joined: 04 Apr 2003
Posts: 412

PostPosted: Mon May 05, 2003 3:53 am    Post subject: Reply with quote

csnyder wrote:
The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though.

Exactly how difficult would it be for you to make a policy of having the server do a daily rsync, and then manually perform it when advisories are released after the daily one? Eight times a day to save a little effort still seems excessive to me. Not to mention that there have been dry spells on the order of nearly a week between GLSA announcements in the past.
_________________
Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy
Back to top
View user's profile Send private message
aethyr
Veteran
Veteran


Joined: 06 Apr 2003
Posts: 1085
Location: NYC

PostPosted: Mon May 05, 2003 3:59 am    Post subject: Reply with quote

I think it's safe to say there's no need to be rsync'ing more than twice a day and I can only hope that those people do it due to being unaware of the situation. Hopefully this will help raise awareness.

I see people on here who incessantly `emerge sync && emerge -u world` without realizing that they're wasting resources.

Gentoo's already bandwidth intensive enough without people hammering the servers...

For anyone interested in further reducing bandwidth on Gentoo servers, you might want to take a peek at this thread a couple weeks ago:

https://forums.gentoo.org/viewtopic.php?t=49139

I wonder how much bandwidth Gentoo consumes per capita versus some other distributions; that might be an interesting statistic.
Back to top
View user's profile Send private message
PowerFactor
Veteran
Veteran


Joined: 30 Jan 2003
Posts: 1693
Location: out of it

PostPosted: Mon May 05, 2003 4:58 am    Post subject: Reply with quote

I agree that to be rsyncing more than twice a day on a regular basis is probably excessive. Realy, what advantage do you get by automatically rsyncing more than once a day. Like the message in the first post said,
Quote:
There's being on the bleeding edge, and there is being just plain silly.
You can always rsync manually when there is a glsa, not like it's hard.

One other thing I see is, judging from conversations on these forums, people are a little too quick to nuke their distfiles in my opinion. Just because it is safe to do doesn't make it a good idea, it's certainly not courteous. By all means, clean out the cruft, but why load the distfiles servers excessively by downloading the same package more than once just because you were lazy. I like the distfile diffs idea. But I think we as users could do a lot better with the current system. And please be very quick to share your distfiles among machines if you have more than one. And copy them over if starting a new install. Little things like that, if we all do it , would help a lot I think. Am I completely off my rocker?
Back to top
View user's profile Send private message
aethyr
Veteran
Veteran


Joined: 06 Apr 2003
Posts: 1085
Location: NYC

PostPosted: Mon May 05, 2003 6:16 am    Post subject: Reply with quote

PowerFactor:

I highly agree, which is why I use the python script found on the second page of this thread:
https://forums.gentoo.org/viewtopic.php?t=3011

This thread is all about cleaning out stale distfiles, which means that the distfiles that are still currently "in use" remain on your system unless they aren't used by any packages. This prevents re-downloading of the same files.

Again, like I've been saying to anyone who will listen, I think the key is to keep these distfiles on the harddrive and use small, incremental patches to patch them to newer versions.

Gentoo can be MUCH more bandwidth friendly (while still being source-based) than it is right now. I hope that the developers are keeping this in mind for the future.
Back to top
View user's profile Send private message
dol-sen
Retired Dev
Retired Dev


Joined: 30 Jun 2002
Posts: 2805
Location: Richmond, BC, Canada

PostPosted: Mon May 05, 2003 6:25 am    Post subject: Reply with quote

I have setup one of my boxes to serve via NSF its /usr/portage dir. It wasn't that hard and seems to work quite well. I just have to finnish some changes to my newer box and all will be using NFS to one /usr/portage. I rsync from 3 to 6 times per week depending what I'm doing or if I'm looking for some update to come thru.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
Lovechild
Advocate
Advocate


Joined: 17 May 2002
Posts: 2858
Location: Århus, Denmark

PostPosted: Mon May 05, 2003 6:35 am    Post subject: Reply with quote

Hell, I break those guidelines daily at times.. but I'm not happy about it

Problems:

a) broken ebuilds
- Sometimes we have broken ebuilds in unstable, and those of us who act as willing testers for Gentoos unstable branch actually need to do emerge rsync a few times a day (also connected to d) at times.

b) incomplete rsyncing
- Rsync fails all the time for me, stalling halfway, thus I hit another server this can't be helped.

c) important updates
- if I see a GLSA notice I rsync... it's that simple.

d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago.

I can only conclude that rsync needs to die, and it has already be discussed in a previous GWN that we need to replace rsync with a less server demanding protocol - cvsup fx. but nothing has happened in the matter AFAIK. I'm known not to be rsync' greatest cheerleader.

Maybe we could get an option to pay for rsync services if we needed regular access, and then give the paying users prior access to mirrors (I'm guessing this can be done using iptables rules and some sort of authentication) - I would, for one, really hate to get the message, "You have reached your daily limit", and then just be kicked off the network.
Back to top
View user's profile Send private message
Lovechild
Advocate
Advocate


Joined: 17 May 2002
Posts: 2858
Location: Århus, Denmark

PostPosted: Mon May 05, 2003 6:38 am    Post subject: Reply with quote

aethyr wrote:
PowerFactor:
Again, like I've been saying to anyone who will listen, I think the key is to keep these distfiles on the harddrive and use small, incremental patches to patch them to newer versions.


I've been talking about this for ages, and the conclusion has always been that it's just to hard to implement. It would be a mighty cool feature, I agree.
Back to top
View user's profile Send private message
aethyr
Veteran
Veteran


Joined: 06 Apr 2003
Posts: 1085
Location: NYC

PostPosted: Mon May 05, 2003 7:06 am    Post subject: Reply with quote

Lovechild, two points:

#1, updating through patching is already being done in other source distributions like sorcerer/sourcery (dunno if sourcemage is doing it), so it can't be impossible (again, I discussed possible implementation strategies, feel free to brainstorm in that thread, I might try to hack a prototype model when I'm finished this semester).

#2, if everyone kept their rsync usage to a reasonable level there'd probably be no need to get denied, or have to pay. I think raising awareness through the GWN, forums, and even on the mirrors is a valid way to get this message across.
Back to top
View user's profile Send private message
Lovechild
Advocate
Advocate


Joined: 17 May 2002
Posts: 2858
Location: Århus, Denmark

PostPosted: Mon May 05, 2003 7:18 am    Post subject: Reply with quote

1) I know - but that don't change the fact that it's hard to do.. I've been looking at how to go about it, and I have a few promising ideas, but the best option might be to simply enforce the use of a "base package" and then release everything from there as diffs, the kernel fx. already does this.

2) I also know that - I would have no problem paying because I know I use the mirrors more than most people.. I was talking to a few people a while back about replacing and Gentoo transfers with BitTorrent to surcome this saturation issue and I think it would be an exellent approach for the file transfers but the rsyncing is a bit trickier to do, a sidebenefit here would be that with bittorrent it would be DEAD easy to setup a mirror, all you would have to do was to start bittorrent and point it to the tracker, as soon as you were done downloading you would automatically become a mirror, and would stay that way untill bittorrent was closed.
Back to top
View user's profile Send private message
aethyr
Veteran
Veteran


Joined: 06 Apr 2003
Posts: 1085
Location: NYC

PostPosted: Mon May 05, 2003 8:19 am    Post subject: Reply with quote

on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).

All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements.
Back to top
View user's profile Send private message
amne
Bodhisattva
Bodhisattva


Joined: 17 Nov 2002
Posts: 6378
Location: Graz / EU

PostPosted: Mon May 05, 2003 8:36 am    Post subject: Reply with quote

one thing about reporting problems with broken mirrors:
i once connected to a rsync mirror that downgraded my portage tree to a >1 week old version, but i did not report this as a bug.
reason: i did not know which mirror it was as this information hat scrolled out of the terminal (and out of its buffer) in the meanwhile. since then i usually try to take a look at the mirror. maybe emerge sync should print the server information at the end and/or log it to /var/log/emerge.log, too.
Back to top
View user's profile Send private message
klieber
Bodhisattva
Bodhisattva


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon May 05, 2003 10:25 am    Post subject: Reply with quote

csnyder wrote:
Another thought...

I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment.

Yes. It is.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
klieber
Bodhisattva
Bodhisattva


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon May 05, 2003 10:26 am    Post subject: Reply with quote

aethyr wrote:
Gentoo can be MUCH more bandwidth friendly (while still being source-based) than it is right now. I hope that the developers are keeping this in mind for the future.

Yes. We are.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
TheCoop
Veteran
Veteran


Joined: 15 Jun 2002
Posts: 1814
Location: Where you least expect it

PostPosted: Mon May 05, 2003 10:33 am    Post subject: Reply with quote

There was some discussion about a p2p file sharing system (volunteers only ofc), so a gentoo install could share one or two files, so most of the main distfile servers could be turned into rsync servers which would help the load somewhat...

Maybe it could be adapted for a p2p rsync, a rsync server on a standard gentoo install that would only allow one connection at a time...
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock
Back to top
View user's profile Send private message
klieber
Bodhisattva
Bodhisattva


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon May 05, 2003 10:39 am    Post subject: Reply with quote

To follow up on the GWN article and clarify a few points:
  • These are guidelines, not rules. We are asking for your help in keeping our rsync mirror system sustainable for the future. That said, if you occasionally have to sync a third time in a 24 hour period, you're not going to have jack-booted thugs show up on your doorstep with a cluebat...
  • Only the most aggregious users will be rate limited. Syncing more than twice a day won't automatically get you rate limited. Syncing against one particular server, 50 times per day almost certainly will get you rate limited. Don't be stupid and you don't have to worry about it.
  • Don't single out particular servers I can't stress this one enough. Not only is it poor etiquette, it's stupid as well -- you draw attention to yourself because your IP address hits the logs of that server over and over. If you use rsync.<continent>.gentoo.org or rsync.<country>.gentoo.org, you'll spread your usage over many servers. This is a Good Thing.
This is a very real problem, folks. One of our German mirror admins ran a script against his logs and found that the top 5 users were consuming over 50% of the CPU resources going to rsync on that machine. (Those users have all been rate limited, btw.) We're not trying to be jerks about this -- but when I have multiple rsync mirror admins complaining to me about the amount of resources we're taking up, we have to do something about it. With your cooperation and assistance, this GWN article will be all we have to do.

One final point to Lovechild's question/suggestion on paid access to rsync mirrors. That's something we're also looking at doing. The issue comes down to building out the payment processing infrastructure. It's on the radar and something that I'd like to see us do. I just don't know when we'll have the time or resources to do so.

--kurt
_________________
The problem with political jokes is that they get elected


Last edited by klieber on Mon May 05, 2003 12:36 pm; edited 1 time in total
Back to top
View user's profile Send private message
klieber
Bodhisattva
Bodhisattva


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon May 05, 2003 10:44 am    Post subject: Reply with quote

Lovechild wrote:
d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago.

Have you considered waiting 4 or even (gasp) 6 hours to let the changes propogate naturally? All the rsync mirrors sync every 30 minutes against one master rsync mirror (rsync1.us.gentoo.org) so all mirrors should have new updates within 30 minutes of them hitting the tree.

That said, I fail to see how waiting a few hours to let the changes work their way down is really that onerous of a request. IMO, the rest of your points were valid -- this one is bogus.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
fishhead
Apprentice
Apprentice


Joined: 07 Mar 2003
Posts: 162
Location: Pasadena, CA

PostPosted: Mon May 05, 2003 11:04 am    Post subject: Reply with quote

Perhaps it would help to have emerge check (by default) how many rsyncs have been done in a 24 hour period and print out a message after the fourth or fifth time. Part of the problem may be that users just aren't THINKING about what they are doing.
Back to top
View user's profile Send private message
Lovechild
Advocate
Advocate


Joined: 17 May 2002
Posts: 2858
Location: Århus, Denmark

PostPosted: Mon May 05, 2003 12:31 pm    Post subject: Reply with quote

aethyr wrote:
on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).

All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements.


Ah, but all the mirrors are still running bittorrent, so the bw. is actually better even when it's an "unpopular" file... like emacs or something :)

Bittorrent would make it a lot easier to cope with sudden rushs - like when the new versions of KDE, xfree and GNOME are released - big popular packages.
Back to top
View user's profile Send private message
Lovechild
Advocate
Advocate


Joined: 17 May 2002
Posts: 2858
Location: Århus, Denmark

PostPosted: Mon May 05, 2003 12:40 pm    Post subject: Reply with quote

One question on the load spreading using set mirrors.

I know I can set GENTOO_MIRRORS in make.conf - but I thought rsync already pointed to a master hub that spread the load to a random non busy mirror.

Anyways if this is not the case, then what's the syntax we need to add, and where's the "official" list of rsync mirrors (I assume we have one someplace) so we can change it to aid the cause.

-edit-

I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm
Back to top
View user's profile Send private message
klieber
Bodhisattva
Bodhisattva


Joined: 17 Apr 2002
Posts: 3657
Location: San Francisco, CA

PostPosted: Mon May 05, 2003 12:52 pm    Post subject: Reply with quote

Lovechild wrote:
I know I can set GENTOO_MIRRORS in make.conf - but I thought rsync already pointed to a master hub that spread the load to a random non busy mirror.

It's not a master hub -- it's handled through DNS via a round robin system.

Lovechild wrote:
I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm

See this Tips and Tricks. You can change your SYNC variable in /etc/make.conf to something that is geographically closer to you. Personally, I encourage all users to use the continent-level rsync rotations.

--kurt
_________________
The problem with political jokes is that they get elected
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2, 3, 4  Next
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum