Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rsync etiquette guideline
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9004
Location: beyond the rim

PostPosted: Mon May 12, 2003 5:34 pm    Post subject: Reply with quote

It's easier than you might think. Setting up the server is creating /etc/exports ("man 5 exports") and "/etc/init.d/nfs start" in most cases. When I first created this setup I thought it would be difficult but it's a lot easier than setting up a samba or apache server.
Back to top
View user's profile Send private message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 737
Location: US-FL-EST

PostPosted: Mon May 12, 2003 6:52 pm    Post subject: Reply with quote

Genone wrote:
It's easier than you might think. Setting up the server is creating /etc/exports ("man 5 exports") and "/etc/init.d/nfs start" in most cases. When I first created this setup I thought it would be difficult but it's a lot easier than setting up a samba or apache server.


Well, there are a few details. :-/

"man 5 exports" doesn't return anything if you don't have nfs-utils emerged.

After emerging nfs-utils, /etc/init.d/ has 'nfs' and 'nfsmount' added to it. I'm not yet clear as to why this is so since 'netmount' seems to mount NFS shares fine. I do see that 'nfs' is required for starting the NFS server though.

I think I've got a handle on what I need to do when I get home.

On the client's side, in addition to what I already posted, I need to update 'hosts.deny' and 'hosts.allow' for 'portmap'.

hosts.deny -- add "portmap:ALL"

hosts.allow -- add "portmap: 192.168.0.0/255.255.255.0

For the server:

1) I need to update hosts.allow & hosts.deny as above and also for 'lockd', 'mountd', 'statd' and 'rquotad'.

2) rc-update add nfs default

3) create /etc/exports with the lines:
Code:

/usr/portage                        192.168.0.0/255.255.255.0(ro)
/usr/portage/distfiles              192.168.0.0/255.255.255.0(rw)
/usr/portage/packages/$HOME         192.168.0.0/255.255.255.0(rw)


I think that is it. I'll find out for sure when I get home tonight. ;-)

Anyone have an opinion if I set this up on my firewall? I have iptables set up _very_ restrictively and basically only permit internet access to my ssh login (root's dissallowed). I realize that this is not pure secure practice, but I'm fairly comfortable that my firewall is pretty secure. It's really the only computer I'm will to leave running 24 hours. It has a 45watt power supply. ;-)
_________________
People whom think M$ is mediocre, don't know the half of it.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9004
Location: beyond the rim

PostPosted: Mon May 12, 2003 7:39 pm    Post subject: Reply with quote

dufeu wrote:
On the client's side, in addition to what I already posted, I need to update 'hosts.deny' and 'hosts.allow' for 'portmap'.

hosts.deny -- add "portmap:ALL"
hosts.allow -- add "portmap: 192.168.0.0/255.255.255.0


wasn't necessary for me, but your setup may differ.

dufeu wrote:
3) create /etc/exports with the lines:
Code:
/usr/portage                        192.168.0.0/255.255.255.0(ro)
/usr/portage/distfiles              192.168.0.0/255.255.255.0(rw)
/usr/portage/packages/$HOME         192.168.0.0/255.255.255.0(rw)


As was said on the mailinglist this won't work, you can't export subdirectories if their parents are already exported. You either have to export /usr/portage RW or move distfiles and packages out of /usr/portage.

dufeu wrote:
It's really the only computer I'm will to leave running 24 hours. It has a 45watt power supply. ;-)


Nice :)
Back to top
View user's profile Send private message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 737
Location: US-FL-EST

PostPosted: Mon May 12, 2003 9:38 pm    Post subject: Reply with quote

Genone wrote:
dufeu wrote:
3) create /etc/exports with the lines:
Code:
c
/usr/portage/distfiles              192.168.0.0/255.255.255.0(rw)
/usr/portage/packages/$HOME         192.168.0.0/255.255.255.0(rw)


As was said on the mailinglist this won't work, you can't export subdirectories if their parents are already exported. You either have to export /usr/portage RW or move distfiles and packages out of /usr/portage.


You are absolutely correct. I read that. I noted it. And I still went ahead and wrote it down wrong. /etc/exports should be:
Code:

/usr/portage                        192.168.0.0/255.255.255.0(rw)


Thanx for catching this. :-)
_________________
People whom think M$ is mediocre, don't know the half of it.
Back to top
View user's profile Send private message
TRauMa
n00b
n00b


Joined: 26 Nov 2002
Posts: 43
Location: Germany

PostPosted: Mon May 12, 2003 11:59 pm    Post subject: Finding out if a package-update is there Reply with quote

Uhm, I hope I just missed it, but there were 4 "I-want-bleding-edge-and-rsync-ten-times-a-day-if-I-wait-for-a-new-version-of-a-package"-guys and no one mentions Gentoo Linux Stable? Check for new versions there, every 60 mins if you like. :roll:
Back to top
View user's profile Send private message
Davin
n00b
n00b


Joined: 13 May 2003
Posts: 45
Location: United States

PostPosted: Tue May 13, 2003 2:21 am    Post subject: Reply with quote

I read this thread in its entirety and the discussion of effective NFS usage in a Gentoo environment jogged my memory a little. This is vaguely off-topic, but it does concern minimalizing Gentoo resources on a network of Gentoo machines.

http://forums.gentoo.org/viewtopic.php?t=53595
Back to top
View user's profile Send private message
hmiller68
Tux's lil' helper
Tux's lil' helper


Joined: 21 Dec 2002
Posts: 77

PostPosted: Thu May 15, 2003 5:56 am    Post subject: Question regarding the use of mirrorselect. Reply with quote

Would I be in the same boat as hardcoding a particular server in config by using the mirrorselect tool in portage ? I mean in actuallity it is infact doing just that isn't ,hardcoding the server in /etc/make.conf to start with the first server selected every time?
Back to top
View user's profile Send private message
really
Guru
Guru


Joined: 27 Aug 2002
Posts: 430
Location: nowhere

PostPosted: Thu May 15, 2003 11:29 am    Post subject: Reply with quote

why is it so important for some people to rsync every hour or so?

i do it when i want to install a new program or update one i see has an update, or if the GLSA shows i really need to update a package..
i rsync like, this week i was i think 3 times (needed to update wine and installed openoffice (gooood i hate it) and checked if something needs updating with emerge -up system..)
last week i didnt rsync at al.. and i think i rsynced once the week before that to see if emerge -up system and world shows something new :)

really unnecesery to have a cronjob rsyncing..
its not like yuu cant live for some days without having the latest of x program.
hell i right now have not yet got gcc3.3. ... gcc3.2 is just fine.. it does it job :D
but i will update in a week or so..

proposal (dont have the time to read the whole threa, maybe someone already proposed it, but):
all the rsyncers could make a site which someone updates every hour with the information on what if anything has been updated/changed. the normal rsyncers could then make a script to look for information on that site. and the site itself gets its information from developers or just one sync..
so all the ppl who feel the need to rsync every x minutes could just look for specific information on a site, not have to sync with the whole tree.
and if they then want to update a program they have, they could just download the ebuild for it from say bugs.gentoo.org if it is there and install it, no need to download the whole tree.
_________________
NoManNoProblem

Get lost before you get shot.
Back to top
View user's profile Send private message
oxo42
n00b
n00b


Joined: 18 May 2003
Posts: 1

PostPosted: Mon May 19, 2003 12:08 am    Post subject: Reply with quote

henke wrote:
Right now I'm wating for xfree-4.3.0-r3 (Castle Rock driver baby :) ) How do I find out if this package has been released without rsyncing?


Okay this is just a temporary measure, but i wrote a php script to browse an rsync server. http://rucus.ru.ac.za/~oxo/rsync/rsync-browser.tar.bz2

Please keep in mind that this was a quick hack for my own benefit and is far from perfect. I also use modlayout so the html will look imcomplete
Back to top
View user's profile Send private message
CRC
Tux's lil' helper
Tux's lil' helper


Joined: 30 Mar 2003
Posts: 90
Location: Dallas, TX, USA

PostPosted: Mon May 24, 2004 11:57 pm    Post subject: Re: Another way to handle sync Reply with quote

Manuzhai wrote:

Is that about right? I would propose an extra layer, then:

1. You get a list (this could be implemented in XML very well) of all the ebuilds currently in Portage. This would include versions as well as the stability for different architectures and dependencies. If the file gets to big, it could be split up into several files for every category. This list will be your primary tree.

2. Ebuilds are only saved on your computer for packages that you actually emerged. This means rsync does not have to be used at all for ebuilds. Some info can be cut from the ebuild because it is already in the XML file.

3. The actual files are retrieved from the internet as stated in the ebuild, and the package is built and compiled.

The advantages:

- Only use rsync (possibly) for the XML files
- Portage tree is much smaller (XML also compresses very well)
- Easy to get info from the tree for emerge as well as external tools
- Less bandwidth used as ebuilds are only transferred as necessary


I think there is some good information here. I think what we need to look at is determining if anything has changed without going through a a full rsync in the entire tree, and limiting the "pull" of ebuilds to just what we are interested in.

Basically, I think you are saying that instead of constant rsync, we do it "on-demand", with a central smaller database file to coordinate it. So, if you emerge say "gaim", we grab the latest unmasked ebuild (which will depend on which fork you use, ie: if you are using ACCEPT_KEYWORDS, plain/regular Gentoo, a break-my-gentoo system, or some "stable-server" off-shoot). Then, you grab that ebuild, then from that information figure out what dependant ebuilds need to be grabbed.

In addition, since many ebuilds are just version bumps, if the system could see that you already have an ebuild for version 0.76 on your system, it could transmit just the diffs between 0.77 and 0.76. Just rename the old ebuild to the new one and rsync - rsync can send diffs, even compressed diffs, automatically.

Also, this means you don't transfer ebuilds for packages you don't care about. An emerge sync would only update the master database of which ebuilds are "current" that simply transmits the diff's from the current version - and a simple timestamp check on that one master would pretty much end needless rsync's quickly! Also, this master could perhaps be indexed like a database to speed up searches. We'd still have people hitting this file often, but see below on why this won't be bad (re: Bit-Torrent to download).

Finally, at this point, your /usr/portage is basically a cache containing only the ebuilds that you've installed - so they can be diff'd with the latest version as an update, and that master database of versions, plus whatever distfiles are still around. If you could share that Bit-Torrent style, that would be really ideal. Say, maybe your system has a minimum time it has to stay on the network for every sync, and stays on the duration of every emerge - contributing to the network during the compilation and installation process, and maybe a few minutes longer. The whole /usr/portage could be shared, distfiles and all. This would use torrent files (or embed the torrent information in the master database) in place of md5 sums. Wanna rsync constantly? Fine! Your donating your resources every time you do, and needless rsyncs just help everyone else out more than they hurt.

If you could do source diffs between versions that would help too, but I think this would be an addition to the ebuild, some option in the file that says that if a particular old distfile exists, then grab the patch instead of a new distfile. Either way, you'd be fetching new distfiles and ebuilds from all the people who currrently have them anyway, and not just the mirrors, so rushes when new versions come out becomes a good thing! More people to share with - exactly the point of bit-torrent.

This would involve some pretty invasive changes to portage, but I think it would be well worth it. Anyone want to try making an "rsync-over-bittorrent" ? How about a central datase of packages that lists current versions of packages and .torrent information for each version (so that you don't need .torrent files maybe, just query the info)? I bet if people did it, or did some proof-of-concept work on it, Gentoo would use it.

All Feedback appreciated! I'm not construction the solution so much as trying to get people thinking about what the requirements are for doing this "the right way".
_________________
Unix/Linux Consulting & Hosting
We Support Gentoo!
http://CoolRunningConcepts.com

Freenode: Taro!
Back to top
View user's profile Send private message
CRC
Tux's lil' helper
Tux's lil' helper


Joined: 30 Mar 2003
Posts: 90
Location: Dallas, TX, USA

PostPosted: Tue May 25, 2004 1:18 am    Post subject: Reply with quote

Just to add - this isn't totally bizarre. Imagine if all the official mirrors supported bittorrent, plus the users ... you'd always have plenty of uploaders.

Look at this post for more info:
http://lists.debian.org/debian-devel/2003/07/msg00473.html

Another possibility would be to look into using PDTP instead of Bit-Torrrent, so you don't have .torrent files and such. This looks like a closer match for the requirement, but isn't nearly ready for general use.
http://pdtp.org/static/index.html
_________________
Unix/Linux Consulting & Hosting
We Support Gentoo!
http://CoolRunningConcepts.com

Freenode: Taro!
Back to top
View user's profile Send private message
salam
Apprentice
Apprentice


Joined: 29 Sep 2005
Posts: 192

PostPosted: Wed Dec 21, 2005 1:44 pm    Post subject: a question about rsync etiquette Reply with quote

this rule is quite easy to understand - the purpose is not to allow excessive load on the servers. but i have a question:
it is clear, that the IP of the abusive user will be dropped in iptables. but what if there are many users? (NAT-ed networks) let's say there are 30 gentoo users and each one syncs 1x/day. none of them broke the etiquette, but the sync mirror sees 30 syncs/day because of NAT. i think it cannot be solved from the server side very well, but what do you think about it? is it a common problem(like on IRC, one abuser gets many other banned)?
Back to top
View user's profile Send private message
certocivitas
Apprentice
Apprentice


Joined: 29 Dec 2004
Posts: 194

PostPosted: Wed Dec 21, 2005 1:50 pm    Post subject: Reply with quote

Most of the time wouldn't the people sharing an IP address be in close proximity to each other. The correct etiquette for Gentoo users in that situation should be to setup a rsync mirror (and http-replicator for that matter). :wink:
Back to top
View user's profile Send private message
aidanjt
Veteran
Veteran


Joined: 20 Feb 2005
Posts: 1101
Location: Rep. of Ireland

PostPosted: Wed Dec 21, 2005 2:04 pm    Post subject: Reply with quote

30 Gentoo users sitting next to each other and none of them had the bright idea of configuring an rsync mirror?.. thats disapointing.
Back to top
View user's profile Send private message
tomk
Administrator
Administrator


Joined: 23 Sep 2003
Posts: 7219
Location: Sat in front of my computer

PostPosted: Wed Dec 21, 2005 2:35 pm    Post subject: Reply with quote

Merged previous three posts.
_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
zialien
n00b
n00b


Joined: 27 Jun 2006
Posts: 31
Location: Melbourne, Australia

PostPosted: Sat Jun 02, 2007 8:23 am    Post subject: just a quick question Reply with quote

I will be converting my XP box desktop to gentoo soon (my laptop is gentoo and has been for well over a year). I am considering making the desktop a local rsync mirror. My question is if i use the instruction on http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror i get the feeling that the files i have in the servers distfiles will not be used, is this assumption correct.
Basically what i am asking is if i have my laptop updating packages will is actually attempt to get the source files from my local server or not. I know that i will get the portage from my mirror but i am not sure about the distfiles. The main reason i ask is because of this line:
exclude = distfiles/ packages/
it makes me think it will not use my mirror for the distfiles which defeats the main purpose of using the local mirror.

and the last question, assuming the above happens (that is my laptop will attempt to get the files from my servers distfiles) if the file does not exist will it download it off a propper mirror and place a copy on my server?

Thanks for any help
Back to top
View user's profile Send private message
mark_alec
Bodhisattva
Bodhisattva


Joined: 11 Sep 2004
Posts: 6066
Location: Melbourne, Australia

PostPosted: Sat Jun 02, 2007 8:45 am    Post subject: Reply with quote

zialien,

A local rsync mirror will only mirror the portage tree, not any distfiles you download. For that purpose, you should look at HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
_________________
www.gentoo.org.au || #gentoo-au
Back to top
View user's profile Send private message
zialien
n00b
n00b


Joined: 27 Jun 2006
Posts: 31
Location: Melbourne, Australia

PostPosted: Sat Jun 02, 2007 9:18 am    Post subject: Reply with quote

thanks for that i saw http-replicator eairler and thought it may be the answer but didnt bother looking anyfurther, my bad. thanks again.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page Previous  1, 2, 3, 4
Page 4 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum