Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Local Rsync Mirror
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Fri Aug 06, 2004 6:54 pm    Post subject: Reply with quote

never heard anything back about the newsletter...
Back to top
View user's profile Send private message
drakos7
Apprentice
Apprentice


Joined: 21 Feb 2003
Posts: 294
Location: Rockville, MD, USA, Earth, Sol

PostPosted: Mon Aug 30, 2004 6:21 pm    Post subject: Reply with quote

Great little HOWTO. I am thinking about providing this to my university as we are now charged per GB of external traffic. Not that syncing transfers much but...
One question from brain dead me: How do I go about getting my new local rsync server to sync to a main mirror on a regular basis. I assume it is accomplished using a cron command, but could someone please send me an actual cron command to do it? My thought is to put this in my crontab file:
Code:
0       0,3,6,9,12,15,18,21       *       *       *       emerge sync  >/dev/null 2>&1
 
So that it syncs every 3 hours (I hope! Is this right?) instead of the primary mirror's 2x/hour.
Is every 3hr too often?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Tue Aug 31, 2004 4:15 pm    Post subject: Reply with quote

Setting up this rsync mirror on a campus is a great idea!! You should also setup a cache for packages if you really want to save some traffic:
http://forums.gentoo.org/viewtopic.php?t=173226

Your users will want to use the cache because it is much faster!

There is only one other thing you must do. The gentoo rsync guidelines at http://www.gentoo.org/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3 suggest syncing more than 1-2 times a day for an individual user is bad. Since your setting up a private rsync mirror for many users, you may need to coordinate your use with an official mirror admin to prevent being banned. If an official mirror admin thinks your an individual user and are abusing the rsync system, you could get your ip banned for a day.

The reason I wrote this howto is because the gentoo rsync mirror doc was out of date and confusing for local rsync mirror configuration. Now that the official doc has been updated, you should also check it: http://www.gentoo.org/doc/en/rsync.xml

Just remember the only section to apply is section 4 - Setting up your own local rsync mirror
Back to top
View user's profile Send private message
drakos7
Apprentice
Apprentice


Joined: 21 Feb 2003
Posts: 294
Location: Rockville, MD, USA, Earth, Sol

PostPosted: Tue Aug 31, 2004 6:26 pm    Post subject: Reply with quote

thx flybynite. I saw your other posting for caching packages while I was looking up this procedure. That is my next step.

As to syncing often, I may move it to every 4 or 5 hours. I am not strictly an "individual user" and the rotation of mirrors helps somewhat. Besides, how often do people really need to sync? Ya know?

It is definitely not as bad as where I found my cron example. The guy had his machine set to sync every hour! :evil: Rather inconsiderate I do believe. And it was posted to these forums! 8O
Back to top
View user's profile Send private message
afabco
Guru
Guru


Joined: 24 Feb 2004
Posts: 380

PostPosted: Thu Sep 30, 2004 8:23 pm    Post subject: Reply with quote

At the risk of missing something idiotically obvious, why would I want to do this, rather than just nfs-mount /usr/portage from client machines to the host?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Fri Oct 01, 2004 11:55 pm    Post subject: Reply with quote

afabco wrote:
At the risk of missing something idiotically obvious, why would I want to do this, rather than just nfs-mount /usr/portage from client machines to the host?


This has been discussed before, look for more info on the boards. Here is the short version....

There are numerous security and other problems including non existent file locking plus nfs is slower than local disk access. Portage can't share some of the directories under /usr/portage without possible corruption problems.

However, If you own/control all machines and/or you totally trust all users of all machines and all others on the network, and you only mount specific directories which aren't machine specific, and you don't edit files that would fubar other machines, and/or you don't dare emerge sync from more than 1 machine at a time and/or you dont dare emerge/download a package from more than 1 machine at a time , and you don't dare etc etc .... it can be done as long as your still comfortable with the remote possibility of file corruption.

The only reason I can think of to nfs share /portage would be if you had a really small disk on a box. As long as you have the disk space, the local rsync mirror is a simple, safe, and the highest preformance option. You can even share it with your friends at a college, work, or gentoo install party.
Back to top
View user's profile Send private message
afabco
Guru
Guru


Joined: 24 Feb 2004
Posts: 380

PostPosted: Sun Oct 03, 2004 3:19 am    Post subject: Reply with quote

OK, that's kinda what I thought. Thank you for replying

Ignoring the fact that I'm very dubious about trusting NFS (usually a lot more trouble than it's worth, and 9 times out of 10 I give up in disgust and use smbmount and then there's the squash flakiness and....and...) :roll:

But when it does work, most of my complaints are handled with the 'soft' mounting option....

Isn't /usr/portage supposed to be identical no matter where it is? Except for /usr/portage/distfiles of course. So the only machine that would ever emerge sync is the hosting machine; all the rest would only do emerge <pkg> or emerge -uD world?

There is the speed thing of course, but of more concern is the >4G disk space requirement (and that's if one aggressively and routinely cleans it out). I -thought- I took care of that by just setting up a 10 G partition on my central machine and then nfs sharing it all around.

I'll have a look around the board for other discussion. Any pointers?
Back to top
View user's profile Send private message
jussen
n00b
n00b


Joined: 11 Apr 2004
Posts: 5

PostPosted: Sun Oct 03, 2004 6:41 pm    Post subject: Reply with quote

i want to link my uni server for emerge but it always fails is it possible that the server only have the files online but not an sync server installed??


Code:
bash-2.05b# emerge sync
>>> starting rsync with rsync://131.246.120.27/pub/linux/gentoo-portage...
>>> checking server timestamp ...
@ERROR: Unknown module 'pub'
rsync: connection unexpectedly closed (41 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry ...
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Wed Oct 06, 2004 2:49 am    Post subject: Reply with quote

afabco wrote:

but of more concern is the >4G disk space requirement (and that's if one aggressively and routinely cleans it out)


The portage tree is really about 600mb. This is the part that is needed by gentoo and what you get when you emerge sync.

All else is space needed only temporarily and can be deleted. Gentoo doesn't do this automatically for various reasons. Many users don't even know what they can delete.

Try it yourself:

Code:

du  -c -h --exclude=distfiles /usr/portage


If you don't have disk space for 600mb on a box then it might be worth the hassle of trying to share /usr/portage and losing portage capabilities now or in the future.

Realize that we can encourage developers to make portage more friendly and work sanely with multiple machines sharing /usr/portage.

There is work being done to make it safe to run two emerge's on one box and were closer today than last year, but for now it is not safe to even run two emerge's on one box - nevermind over a nfs share.
Back to top
View user's profile Send private message
micmac
l33t
l33t


Joined: 28 Nov 2003
Posts: 989

PostPosted: Fri Oct 29, 2004 4:30 pm    Post subject: Reply with quote

Thanks man! Your howto is most useful! Great idea!

Laters
mic
Back to top
View user's profile Send private message
cbradney
Apprentice
Apprentice


Joined: 08 May 2003
Posts: 245
Location: Luxembourg

PostPosted: Fri Oct 29, 2004 5:49 pm    Post subject: Reply with quote

Is this still applicable now? Portage and rsync have been updated since June. IE, is the ebuild for the rsync server solely for a non local network?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Fri Oct 29, 2004 8:19 pm    Post subject: Reply with quote

Good question.... Could you install it and compare it to this howto?

I don't know if the ebuild is still for official mirrors. I did look at the local rsync howto at:
http://www.gentoo.org/doc/en/rsync.xml

and it is much improved.
Back to top
View user's profile Send private message
cbradney
Apprentice
Apprentice


Joined: 08 May 2003
Posts: 245
Location: Luxembourg

PostPosted: Fri Oct 29, 2004 8:26 pm    Post subject: Reply with quote

I've followed the gentoo rsync server howto and it seems to work just fine here. Theres a section at the bottom for local LAN stuff.

Now.. I've also posted on the wiki talk page re the http replicator.. i get an error:
readline43-004
readline43-005

Checking authenticity and integrity of new files...
Searching for ebuilds's ....
Done!

Found 16435 ebuilds.

Extracting the checksums....
Missing digest: net-mail/gml-0.5
Done!

Verifying checksum's....
/usr/portage/distfiles/gdbm-1.8.0.tar.gz
Traceback (most recent call last):
File "/usr/bin/repcacheman", line 198, in ?
if t[0]:
KeyError: 0
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Fri Oct 29, 2004 11:07 pm    Post subject: Reply with quote

fixed:
download an updated repcacheman:
http://www.updatedlinux.com/replicator/portagefix/repcacheman

see the original thread for the background and the fix instructions:

http://forums.gentoo.org/viewtopic.php?t=173226&start=175
Back to top
View user's profile Send private message
cbradney
Apprentice
Apprentice


Joined: 08 May 2003
Posts: 245
Location: Luxembourg

PostPosted: Fri Oct 29, 2004 11:47 pm    Post subject: Reply with quote

Hey.. cool! Seems to be working having done an emerge -f of a package! Now all I need is something I really want to emerge :)

Thanks heaps for this.. any idea about getting it into portage?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Oct 30, 2004 6:39 pm    Post subject: Reply with quote

The ebuild has been submitted. I don't know any reason for the delay....
Back to top
View user's profile Send private message
anderlin
Tux's lil' helper
Tux's lil' helper


Joined: 29 Nov 2003
Posts: 149
Location: Trondheim, Norway

PostPosted: Wed Nov 03, 2004 6:07 pm    Post subject: Reply with quote

I have a rsync-mirror on my LAN, set up following the instructions in this thread, and I want to include some extra ebuilds not in portage. How can that be done?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Wed Nov 03, 2004 7:14 pm    Post subject: Reply with quote

Good question...

I don't know of any portage feature that will help you do this. Doesn't mean it doesn't exist...


You could define another rsync source dir for /usr/local/portage so any machine could separately sync your local ebuilds to their /usr/local/portage....
Back to top
View user's profile Send private message
anderlin
Tux's lil' helper
Tux's lil' helper


Joined: 29 Nov 2003
Posts: 149
Location: Trondheim, Norway

PostPosted: Wed Nov 03, 2004 11:06 pm    Post subject: Reply with quote

That's a possible solution. Another is to replace (alias?) emerge sync on the server with a command that syncs the tree, for then to copy the extra ebuilds into it. Both solutions work, but I don't find them very elegant... Some built in feature would have been nice.
Back to top
View user's profile Send private message
packman
n00b
n00b


Joined: 14 Oct 2002
Posts: 28
Location: Belgium

PostPosted: Fri Nov 05, 2004 2:29 pm    Post subject: Reply with quote

Marwin wrote:
Take your samba-server and make a directory that you call 'distfiles'.
Share it and make the clients mount it at /usr/portage/distfiles.
And Wolla! You've got a shared distfiles :-)


I can assure you - you'll lose a lot of speed with this. Building the dependency cache on a lot of small files on a network drive? Not a good idea imho :)

For distfiles - that's smth else (however, I just use squid for this with a rather large cache ;))
_________________
I heard if you play the XP CD backwards, you get a satanic message. Thats nothing compared to what happens if you play it forward - then it installs Windows XP
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Nov 06, 2004 6:23 pm    Post subject: Reply with quote

I've trie squid a while ago. The problems I had include:

1. Squid names files "lskajdifoa98834o53l2kdsl" so they are not usable by another program.

2. Squid isn't made for binary files. One example is http://gentoo.oregonstate.edu/distfiles/kde-3.3.1.tar.bz2 is not the same to squid as http://some.other.mirror/distfiles/kde-3.3.1.tar.bz2 even though it is the same file.....

3. Squid wants to remove cached files by time. Problem is kde-3.3.1.tar.bz2 is current for months/years? till kde is upgraded, squid still wants to remove files by time not version.

4. You can't add files to the cache yourself - like from an install CD...

I could go on and on...

Just try http-replicator....

http://forums.gentoo.org/viewtopic.php?t=173226
Back to top
View user's profile Send private message
Boris27
Guru
Guru


Joined: 05 Nov 2003
Posts: 562
Location: Almelo, The Netherlands

PostPosted: Sat Nov 06, 2004 6:53 pm    Post subject: Reply with quote

I have a portage server for my local lan.

It's a rsync mirror (with the help of this thread), and exports its distfiles via samba. Distfile sharing like that is pretty nice, if you control all the systems.

If you don't, flakyness can occur. Say you're downloading kdebase-3.3.1 and someone else wants to install it too. Their portage client thinks the file is there (it exists), and starts extracting the partial file. It will fail. Not nice. If you update all the boxes, you know what is downloading where, so no duplicate installs are started.
_________________
we are microsoft, lower your firewalls and surrender your pc's. we will add your biological and technological distinctiveness to our own. your culture will adapt and service us. resistance is futile.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Wed Nov 10, 2004 10:25 pm    Post subject: Reply with quote

Boris27 wrote:
Distfile sharing like that is pretty nice, if you control all the systems.
If you don't, flakyness can occur.


The question is why would you make your system flaky and slow? Yes, if you always remember exactly how to do updates and portage never changes your probably OK. A better solution exists...

Using http-replicator is fast and bulletproof!! You can update ALL your boxes at the same time or at any time without worry and do so knowing that they will finish faster!!

http://forums.gentoo.org/viewtopic.php?t=173226
Back to top
View user's profile Send private message
Gentree
Watchman
Watchman


Joined: 01 Jul 2003
Posts: 5227
Location: France, Old Europe

PostPosted: Fri Dec 17, 2004 3:16 am    Post subject: Reply with quote

I have got my second box to sync off the main system but I want to rsync my overlay files as well.

I added a [overlay] section to /etc/rsync/rsyncd.conf that seems correct:

Code:
rsync rsync://linbox/overlay


this returns a list of the directories on linbox.

Code:
rsync rsync://linbox/overlay/* .


This only tries some of the files and fails with permission errors.

Why does this work on one structure but not on the other?

Thanks for any advice.

8)
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.

AthlonXP-M on A7N8X @ 2.6/2.4GHz (winter/summer)
2.6.32-hh1 : portage ~x86
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Dec 19, 2004 7:14 pm    Post subject: Reply with quote

Gentree wrote:
I have got my second box to sync off the main system but I want to rsync my overlay files as well.

I added a [overlay] section to /etc/rsync/rsyncd.conf that seems correct:

Code:
rsync rsync://linbox/overlay


this returns a list of the directories on linbox.

Code:
rsync rsync://linbox/overlay/* .


This only tries some of the files and fails with permission errors.

Why does this work on one structure but not on the other?

Thanks for any advice.

8)


Here is what I can think of at the moment.

You have to deal with both client and server levels of rsync and portage.

Rsync SERVER has it's own set of default security features and options, such as --safe-links that my init script uses. If you don't use my init script what options does your sever init script use?

The rsync CLIENT has options set by portage. You'll have to dig through portage to find them. Those options do things like skip certain files and ignore certain dir's.

The options could be the cause, or it could just be the file permissions?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6  Next
Page 3 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum