Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
GF15: Rsync etiquette OR How often should I sync?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Frequently Asked Questions
View previous topic :: View next topic  
Author Message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 16102
Location: Colorado

PostPosted: Fri Feb 06, 2004 6:34 pm    Post subject: GF15: Rsync etiquette OR How often should I sync? Reply with quote

Navigation: [Gentoo Fundamentals] [Table of Contents]

Rsync etiquette guidelines

The continued growth of Gentoo Linux has placed more and more demands on our mirror system. Both source mirrors as well as rsync mirrors continue to show dramatic increases in usage. As we continue to grow, the importance of using our mirroring system responsibly becomes more critical. As such, here are some rsync etiquette guidelines to keep in mind as you emerge sync:
  • Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system.
  • Use the rotations, not individual servers Please do not single out specific rsync mirrors and hard-code them in your configuration files. Doing so places undue stress on particular mirrors. By using country- or continent-specific rotations, you are able to select servers that are geographically close to you, yet still distribute the load over a number of servers.
  • Report bad mirrors on bugs.gentoo.org If you notice a server in the rotation that is not responding or is showing signs of other problems, please report it on bugs.gentoo.org immediately. Many users simply assume that someone else will file the bug report, which results in nobody ever finding a bug report until a Gentoo developer happens to notice the problem. By filing bug reports sooner, we can catch and remove problematic mirrors from our rotation much faster.
Remember that all of our rsync mirrors rely entirely upon donated resources. Thus, being respectful of these donated resources is not only a common courtesy, but essential if we are to support the continued growth of Gentoo Linux.


Followup and Clarification

To follow up on the GWN article and clarify a few points:
  • These are guidelines, not rules. We are asking for your help in keeping our rsync mirror system sustainable for the future. That said, if you occasionally have to sync a third time in a 24 hour period, you're not going to have jack-booted thugs show up on your doorstep with a cluebat...
  • Only the most aggregious users will be rate limited. Syncing more than twice a day won't automatically get you rate limited. Syncing against one particular server, 50 times per day almost certainly will get you rate limited. Don't be stupid and you don't have to worry about it.
  • Don't single out particular servers I can't stress this one enough. Not only is it poor etiquette, it's stupid as well -- you draw attention to yourself because your IP address hits the logs of that server over and over. If you use rsync.<continent>.gentoo.org or rsync.<country>.gentoo.org, you'll spread your usage over many servers. This is a Good Thing.
This is a very real problem, folks. One of our German mirror admins ran a script against his logs and found that the top 5 users were consuming over 50% of the CPU resources going to rsync on that machine. (Those users have all been rate limited, btw.) We're not trying to be jerks about this -- but when I have multiple rsync mirror admins complaining to me about the amount of resources we're taking up, we have to do something about it. With your cooperation and assistance, this GWN article will be all we have to do.
_________________
lolgov. 'cause where we're going, you don't have civil liberties.

In Loving Memory
1787 - 2008
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Frequently Asked Questions All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum