View previous topic :: View next topic |
Author |
Message |
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Aug 06, 2004 6:54 pm Post subject: |
|
|
never heard anything back about the newsletter... |
|
Back to top |
|
|
drakos7 Apprentice
Joined: 21 Feb 2003 Posts: 294 Location: Rockville, MD, USA, Earth, Sol
|
Posted: Mon Aug 30, 2004 6:21 pm Post subject: |
|
|
Great little HOWTO. I am thinking about providing this to my university as we are now charged per GB of external traffic. Not that syncing transfers much but...
One question from brain dead me: How do I go about getting my new local rsync server to sync to a main mirror on a regular basis. I assume it is accomplished using a cron command, but could someone please send me an actual cron command to do it? My thought is to put this in my crontab file:
Code: | 0 0,3,6,9,12,15,18,21 * * * emerge sync >/dev/null 2>&1
| So that it syncs every 3 hours (I hope! Is this right?) instead of the primary mirror's 2x/hour.
Is every 3hr too often? |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Aug 31, 2004 4:15 pm Post subject: |
|
|
Setting up this rsync mirror on a campus is a great idea!! You should also setup a cache for packages if you really want to save some traffic:
https://forums.gentoo.org/viewtopic.php?t=173226
Your users will want to use the cache because it is much faster!
There is only one other thing you must do. The gentoo rsync guidelines at http://www.gentoo.org/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3 suggest syncing more than 1-2 times a day for an individual user is bad. Since your setting up a private rsync mirror for many users, you may need to coordinate your use with an official mirror admin to prevent being banned. If an official mirror admin thinks your an individual user and are abusing the rsync system, you could get your ip banned for a day.
The reason I wrote this howto is because the gentoo rsync mirror doc was out of date and confusing for local rsync mirror configuration. Now that the official doc has been updated, you should also check it: http://www.gentoo.org/doc/en/rsync.xml
Just remember the only section to apply is section 4 - Setting up your own local rsync mirror |
|
Back to top |
|
|
drakos7 Apprentice
Joined: 21 Feb 2003 Posts: 294 Location: Rockville, MD, USA, Earth, Sol
|
Posted: Tue Aug 31, 2004 6:26 pm Post subject: |
|
|
thx flybynite. I saw your other posting for caching packages while I was looking up this procedure. That is my next step.
As to syncing often, I may move it to every 4 or 5 hours. I am not strictly an "individual user" and the rotation of mirrors helps somewhat. Besides, how often do people really need to sync? Ya know?
It is definitely not as bad as where I found my cron example. The guy had his machine set to sync every hour! Rather inconsiderate I do believe. And it was posted to these forums! |
|
Back to top |
|
|
afabco Guru
Joined: 24 Feb 2004 Posts: 380
|
Posted: Thu Sep 30, 2004 8:23 pm Post subject: |
|
|
At the risk of missing something idiotically obvious, why would I want to do this, rather than just nfs-mount /usr/portage from client machines to the host? |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Oct 01, 2004 11:55 pm Post subject: |
|
|
afabco wrote: | At the risk of missing something idiotically obvious, why would I want to do this, rather than just nfs-mount /usr/portage from client machines to the host? |
This has been discussed before, look for more info on the boards. Here is the short version....
There are numerous security and other problems including non existent file locking plus nfs is slower than local disk access. Portage can't share some of the directories under /usr/portage without possible corruption problems.
However, If you own/control all machines and/or you totally trust all users of all machines and all others on the network, and you only mount specific directories which aren't machine specific, and you don't edit files that would fubar other machines, and/or you don't dare emerge sync from more than 1 machine at a time and/or you dont dare emerge/download a package from more than 1 machine at a time , and you don't dare etc etc .... it can be done as long as your still comfortable with the remote possibility of file corruption.
The only reason I can think of to nfs share /portage would be if you had a really small disk on a box. As long as you have the disk space, the local rsync mirror is a simple, safe, and the highest preformance option. You can even share it with your friends at a college, work, or gentoo install party. |
|
Back to top |
|
|
afabco Guru
Joined: 24 Feb 2004 Posts: 380
|
Posted: Sun Oct 03, 2004 3:19 am Post subject: |
|
|
OK, that's kinda what I thought. Thank you for replying
Ignoring the fact that I'm very dubious about trusting NFS (usually a lot more trouble than it's worth, and 9 times out of 10 I give up in disgust and use smbmount and then there's the squash flakiness and....and...)
But when it does work, most of my complaints are handled with the 'soft' mounting option....
Isn't /usr/portage supposed to be identical no matter where it is? Except for /usr/portage/distfiles of course. So the only machine that would ever emerge sync is the hosting machine; all the rest would only do emerge <pkg> or emerge -uD world?
There is the speed thing of course, but of more concern is the >4G disk space requirement (and that's if one aggressively and routinely cleans it out). I -thought- I took care of that by just setting up a 10 G partition on my central machine and then nfs sharing it all around.
I'll have a look around the board for other discussion. Any pointers? |
|
Back to top |
|
|
jussen n00b
Joined: 11 Apr 2004 Posts: 5
|
Posted: Sun Oct 03, 2004 6:41 pm Post subject: |
|
|
i want to link my uni server for emerge but it always fails is it possible that the server only have the files online but not an sync server installed??
Code: | bash-2.05b# emerge sync
>>> starting rsync with rsync://131.246.120.27/pub/linux/gentoo-portage...
>>> checking server timestamp ...
@ERROR: Unknown module 'pub'
rsync: connection unexpectedly closed (41 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry ...
|
|
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Wed Oct 06, 2004 2:49 am Post subject: |
|
|
afabco wrote: |
but of more concern is the >4G disk space requirement (and that's if one aggressively and routinely cleans it out) |
The portage tree is really about 600mb. This is the part that is needed by gentoo and what you get when you emerge sync.
All else is space needed only temporarily and can be deleted. Gentoo doesn't do this automatically for various reasons. Many users don't even know what they can delete.
Try it yourself:
Code: |
du -c -h --exclude=distfiles /usr/portage
|
If you don't have disk space for 600mb on a box then it might be worth the hassle of trying to share /usr/portage and losing portage capabilities now or in the future.
Realize that we can encourage developers to make portage more friendly and work sanely with multiple machines sharing /usr/portage.
There is work being done to make it safe to run two emerge's on one box and were closer today than last year, but for now it is not safe to even run two emerge's on one box - nevermind over a nfs share. |
|
Back to top |
|
|
micmac l33t
Joined: 28 Nov 2003 Posts: 996
|
Posted: Fri Oct 29, 2004 4:30 pm Post subject: |
|
|
Thanks man! Your howto is most useful! Great idea!
Laters
mic |
|
Back to top |
|
|
cbradney Apprentice
Joined: 08 May 2003 Posts: 245 Location: Luxembourg
|
Posted: Fri Oct 29, 2004 5:49 pm Post subject: |
|
|
Is this still applicable now? Portage and rsync have been updated since June. IE, is the ebuild for the rsync server solely for a non local network? |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Oct 29, 2004 8:19 pm Post subject: |
|
|
Good question.... Could you install it and compare it to this howto?
I don't know if the ebuild is still for official mirrors. I did look at the local rsync howto at:
http://www.gentoo.org/doc/en/rsync.xml
and it is much improved. |
|
Back to top |
|
|
cbradney Apprentice
Joined: 08 May 2003 Posts: 245 Location: Luxembourg
|
Posted: Fri Oct 29, 2004 8:26 pm Post subject: |
|
|
I've followed the gentoo rsync server howto and it seems to work just fine here. Theres a section at the bottom for local LAN stuff.
Now.. I've also posted on the wiki talk page re the http replicator.. i get an error:
readline43-004
readline43-005
Checking authenticity and integrity of new files...
Searching for ebuilds's ....
Done!
Found 16435 ebuilds.
Extracting the checksums....
Missing digest: net-mail/gml-0.5
Done!
Verifying checksum's....
/usr/portage/distfiles/gdbm-1.8.0.tar.gz
Traceback (most recent call last):
File "/usr/bin/repcacheman", line 198, in ?
if t[0]:
KeyError: 0 |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
|
Back to top |
|
|
cbradney Apprentice
Joined: 08 May 2003 Posts: 245 Location: Luxembourg
|
Posted: Fri Oct 29, 2004 11:47 pm Post subject: |
|
|
Hey.. cool! Seems to be working having done an emerge -f of a package! Now all I need is something I really want to emerge
Thanks heaps for this.. any idea about getting it into portage? |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Sat Oct 30, 2004 6:39 pm Post subject: |
|
|
The ebuild has been submitted. I don't know any reason for the delay.... |
|
Back to top |
|
|
anderlin Tux's lil' helper
Joined: 29 Nov 2003 Posts: 149 Location: Trondheim, Norway
|
Posted: Wed Nov 03, 2004 6:07 pm Post subject: |
|
|
I have a rsync-mirror on my LAN, set up following the instructions in this thread, and I want to include some extra ebuilds not in portage. How can that be done? |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Wed Nov 03, 2004 7:14 pm Post subject: |
|
|
Good question...
I don't know of any portage feature that will help you do this. Doesn't mean it doesn't exist...
You could define another rsync source dir for /usr/local/portage so any machine could separately sync your local ebuilds to their /usr/local/portage.... |
|
Back to top |
|
|
anderlin Tux's lil' helper
Joined: 29 Nov 2003 Posts: 149 Location: Trondheim, Norway
|
Posted: Wed Nov 03, 2004 11:06 pm Post subject: |
|
|
That's a possible solution. Another is to replace (alias?) emerge sync on the server with a command that syncs the tree, for then to copy the extra ebuilds into it. Both solutions work, but I don't find them very elegant... Some built in feature would have been nice. |
|
Back to top |
|
|
packman n00b
Joined: 14 Oct 2002 Posts: 28 Location: Belgium
|
Posted: Fri Nov 05, 2004 2:29 pm Post subject: |
|
|
Marwin wrote: | Take your samba-server and make a directory that you call 'distfiles'.
Share it and make the clients mount it at /usr/portage/distfiles.
And Wolla! You've got a shared distfiles |
I can assure you - you'll lose a lot of speed with this. Building the dependency cache on a lot of small files on a network drive? Not a good idea imho
For distfiles - that's smth else (however, I just use squid for this with a rather large cache ) _________________ I heard if you play the XP CD backwards, you get a satanic message. Thats nothing compared to what happens if you play it forward - then it installs Windows XP |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
|
Back to top |
|
|
Boris27 Guru
Joined: 05 Nov 2003 Posts: 562 Location: Almelo, The Netherlands
|
Posted: Sat Nov 06, 2004 6:53 pm Post subject: |
|
|
I have a portage server for my local lan.
It's a rsync mirror (with the help of this thread), and exports its distfiles via samba. Distfile sharing like that is pretty nice, if you control all the systems.
If you don't, flakyness can occur. Say you're downloading kdebase-3.3.1 and someone else wants to install it too. Their portage client thinks the file is there (it exists), and starts extracting the partial file. It will fail. Not nice. If you update all the boxes, you know what is downloading where, so no duplicate installs are started. _________________ we are microsoft, lower your firewalls and surrender your pc's. we will add your biological and technological distinctiveness to our own. your culture will adapt and service us. resistance is futile. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Wed Nov 10, 2004 10:25 pm Post subject: |
|
|
Boris27 wrote: | Distfile sharing like that is pretty nice, if you control all the systems.
If you don't, flakyness can occur. |
The question is why would you make your system flaky and slow? Yes, if you always remember exactly how to do updates and portage never changes your probably OK. A better solution exists...
Using http-replicator is fast and bulletproof!! You can update ALL your boxes at the same time or at any time without worry and do so knowing that they will finish faster!!
https://forums.gentoo.org/viewtopic.php?t=173226 |
|
Back to top |
|
|
Gentree Watchman
Joined: 01 Jul 2003 Posts: 5350 Location: France, Old Europe
|
Posted: Fri Dec 17, 2004 3:16 am Post subject: |
|
|
I have got my second box to sync off the main system but I want to rsync my overlay files as well.
I added a [overlay] section to /etc/rsync/rsyncd.conf that seems correct:
Code: | rsync rsync://linbox/overlay |
this returns a list of the directories on linbox.
Code: | rsync rsync://linbox/overlay/* . |
This only tries some of the files and fails with permission errors.
Why does this work on one structure but not on the other?
Thanks for any advice.
_________________ Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86 |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Sun Dec 19, 2004 7:14 pm Post subject: |
|
|
Gentree wrote: | I have got my second box to sync off the main system but I want to rsync my overlay files as well.
I added a [overlay] section to /etc/rsync/rsyncd.conf that seems correct:
Code: | rsync rsync://linbox/overlay |
this returns a list of the directories on linbox.
Code: | rsync rsync://linbox/overlay/* . |
This only tries some of the files and fails with permission errors.
Why does this work on one structure but not on the other?
Thanks for any advice.
|
Here is what I can think of at the moment.
You have to deal with both client and server levels of rsync and portage.
Rsync SERVER has it's own set of default security features and options, such as --safe-links that my init script uses. If you don't use my init script what options does your sever init script use?
The rsync CLIENT has options set by portage. You'll have to dig through portage to find them. Those options do things like skip certain files and ignore certain dir's.
The options could be the cause, or it could just be the file permissions? |
|
Back to top |
|
|
|