rsync etiquette guideline

csnyder · Posted: Mon May 05, 2003 3:28 am Post subject:

Another thought...

I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment.

shadow255 · Guru Joined: 04 Apr 2003 Posts: 412

shadow255 · Guru Joined: 04 Apr 2003 Posts: 412

csnyder · Posted: Mon May 05, 2003 3:44 am Post subject:

The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though.

shadow255 · Guru Joined: 04 Apr 2003 Posts: 412

aethyr · Veteran Joined: 06 Apr 2003 Posts: 1085 Location: NYC

I think it's safe to say there's no need to be rsync'ing more than twice a day and I can only hope that those people do it due to being unaware of the situation. Hopefully this will help raise awareness.

I see people on here who incessantly `emerge sync && emerge -u world` without realizing that they're wasting resources.

Gentoo's already bandwidth intensive enough without people hammering the servers...

For anyone interested in further reducing bandwidth on Gentoo servers, you might want to take a peek at this thread a couple weeks ago:

https://forums.gentoo.org/viewtopic.php?t=49139

I wonder how much bandwidth Gentoo consumes per capita versus some other distributions; that might be an interesting statistic.

PowerFactor · Posted: Mon May 05, 2003 4:58 am Post subject:

I agree that to be rsyncing more than twice a day on a regular basis is probably excessive. Realy, what advantage do you get by automatically rsyncing more than once a day. Like the message in the first post said,

aethyr · Veteran Joined: 06 Apr 2003 Posts: 1085 Location: NYC

PowerFactor:

I highly agree, which is why I use the python script found on the second page of this thread:
https://forums.gentoo.org/viewtopic.php?t=3011

This thread is all about cleaning out stale distfiles, which means that the distfiles that are still currently "in use" remain on your system unless they aren't used by any packages. This prevents re-downloading of the same files.

Again, like I've been saying to anyone who will listen, I think the key is to keep these distfiles on the harddrive and use small, incremental patches to patch them to newer versions.

Gentoo can be MUCH more bandwidth friendly (while still being source-based) than it is right now. I hope that the developers are keeping this in mind for the future.

dol-sen · Posted: Mon May 05, 2003 6:25 am Post subject:

I have setup one of my boxes to serve via NSF its /usr/portage dir. It wasn't that hard and seems to work quite well. I just have to finnish some changes to my newer box and all will be using NFS to one /usr/portage. I rsync from 3 to 6 times per week depending what I'm doing or if I'm looking for some update to come thru.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

Lovechild · Posted: Mon May 05, 2003 6:35 am Post subject:

Hell, I break those guidelines daily at times.. but I'm not happy about it

Problems:

a) broken ebuilds
- Sometimes we have broken ebuilds in unstable, and those of us who act as willing testers for Gentoos unstable branch actually need to do emerge rsync a few times a day (also connected to d) at times.

b) incomplete rsyncing
- Rsync fails all the time for me, stalling halfway, thus I hit another server this can't be helped.

c) important updates
- if I see a GLSA notice I rsync... it's that simple.

d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago.

I can only conclude that rsync needs to die, and it has already be discussed in a previous GWN that we need to replace rsync with a less server demanding protocol - cvsup fx. but nothing has happened in the matter AFAIK. I'm known not to be rsync' greatest cheerleader.

Maybe we could get an option to pay for rsync services if we needed regular access, and then give the paying users prior access to mirrors (I'm guessing this can be done using iptables rules and some sort of authentication) - I would, for one, really hate to get the message, "You have reached your daily limit", and then just be kicked off the network.

Lovechild · Posted: Mon May 05, 2003 6:38 am Post subject:

aethyr · Veteran Joined: 06 Apr 2003 Posts: 1085 Location: NYC

Lovechild, two points:

#1, updating through patching is already being done in other source distributions like sorcerer/sourcery (dunno if sourcemage is doing it), so it can't be impossible (again, I discussed possible implementation strategies, feel free to brainstorm in that thread, I might try to hack a prototype model when I'm finished this semester).

#2, if everyone kept their rsync usage to a reasonable level there'd probably be no need to get denied, or have to pay. I think raising awareness through the GWN, forums, and even on the mirrors is a valid way to get this message across.

Lovechild · Posted: Mon May 05, 2003 7:18 am Post subject:

1) I know - but that don't change the fact that it's hard to do.. I've been looking at how to go about it, and I have a few promising ideas, but the best option might be to simply enforce the use of a "base package" and then release everything from there as diffs, the kernel fx. already does this.

2) I also know that - I would have no problem paying because I know I use the mirrors more than most people.. I was talking to a few people a while back about replacing and Gentoo transfers with BitTorrent to surcome this saturation issue and I think it would be an exellent approach for the file transfers but the rsyncing is a bit trickier to do, a sidebenefit here would be that with bittorrent it would be DEAD easy to setup a mirror, all you would have to do was to start bittorrent and point it to the tracker, as soon as you were done downloading you would automatically become a mirror, and would stay that way untill bittorrent was closed.

aethyr · Veteran Joined: 06 Apr 2003 Posts: 1085 Location: NYC

on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).

All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements.

amne · Posted: Mon May 05, 2003 8:36 am Post subject:

one thing about reporting problems with broken mirrors:
i once connected to a rsync mirror that downgraded my portage tree to a >1 week old version, but i did not report this as a bug.
reason: i did not know which mirror it was as this information hat scrolled out of the terminal (and out of its buffer) in the meanwhile. since then i usually try to take a look at the mirror. maybe emerge sync should print the server information at the end and/or log it to /var/log/emerge.log, too.

klieber · Posted: Mon May 05, 2003 10:25 am Post subject:

klieber · Posted: Mon May 05, 2003 10:26 am Post subject:

TheCoop · Posted: Mon May 05, 2003 10:33 am Post subject:

There was some discussion about a p2p file sharing system (volunteers only ofc), so a gentoo install could share one or two files, so most of the main distfile servers could be turned into rsync servers which would help the load somewhat...

Maybe it could be adapted for a p2p rsync, a rsync server on a standard gentoo install that would only allow one connection at a time...
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock

klieber · Posted: Mon May 05, 2003 10:39 am Post subject:

To follow up on the GWN article and clarify a few points:

These are guidelines, not rules. We are asking for your help in keeping our rsync mirror system sustainable for the future. That said, if you occasionally have to sync a third time in a 24 hour period, you're not going to have jack-booted thugs show up on your doorstep with a cluebat...
Only the most aggregious users will be rate limited. Syncing more than twice a day won't automatically get you rate limited. Syncing against one particular server, 50 times per day almost certainly will get you rate limited. Don't be stupid and you don't have to worry about it.
Don't single out particular servers I can't stress this one enough. Not only is it poor etiquette, it's stupid as well -- you draw attention to yourself because your IP address hits the logs of that server over and over. If you use rsync.<continent>.gentoo.org or rsync.<country>.gentoo.org, you'll spread your usage over many servers. This is a Good Thing.

This is a very real problem, folks. One of our German mirror admins ran a script against his logs and found that the top 5 users were consuming over 50% of the CPU resources going to rsync on that machine. (Those users have all been rate limited, btw.) We're not trying to be jerks about this -- but when I have multiple rsync mirror admins complaining to me about the amount of resources we're taking up, we have to do something about it. With your cooperation and assistance, this GWN article will be all we have to do.

One final point to Lovechild's question/suggestion on paid access to rsync mirrors. That's something we're also looking at doing. The issue comes down to building out the payment processing infrastructure. It's on the radar and something that I'd like to see us do. I just don't know when we'll have the time or resources to do so.

--kurt
_________________
The problem with political jokes is that they get elected

klieber · Posted: Mon May 05, 2003 10:44 am Post subject:

fishhead · Posted: Mon May 05, 2003 11:04 am Post subject:

Perhaps it would help to have emerge check (by default) how many rsyncs have been done in a 24 hour period and print out a message after the fourth or fifth time. Part of the problem may be that users just aren't THINKING about what they are doing.

Lovechild · Posted: Mon May 05, 2003 12:31 pm Post subject:

Lovechild · Posted: Mon May 05, 2003 12:40 pm Post subject:

One question on the load spreading using set mirrors.

I know I can set GENTOO_MIRRORS in make.conf - but I thought rsync already pointed to a master hub that spread the load to a random non busy mirror.

Anyways if this is not the case, then what's the syntax we need to add, and where's the "official" list of rsync mirrors (I assume we have one someplace) so we can change it to aid the cause.

-edit-

I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm

klieber · Posted: Mon May 05, 2003 12:52 pm Post subject: