HOWTO: Central Gentoo Mirror for your Internal Network

Russel-Athletic · n00b Joined: 07 Jan 2005 Posts: 47

I have one question.
I want the solution with nfs share (only 2 clients) but i don't know which direcotries i have to share. i want to do a emerge sync only on the server maschine and emerge -u world manually.
Ok there is /usr/portage and /usr/portage/distfiles but something else? Something in /var ?

jkroon · Posted: Tue Mar 22, 2005 9:45 pm Post subject:

You see. That is why we have forums. I haven't read that particular man page in a long time. That'll solve part of the problem. Satisfactorily (spelling ...) for now. And probably for a long time to come.

Next question, we maintain some of our own ebuilds, some of these are queued for inclusion or we're pushing to get them in, but some of these will never get included. We have a seperate rsync section for this called gentoo-local-portage to which we can manually rsync. Since we are running the sync's out of cron'ed scripts I suppose I can just add an additional rsync command to those scripts, but it would be nice if portage has some way of doing this for me, ie, if I can ask it to sync /usr/local/portage against rsync://some.server/gentoo-local-portage whenever emerge sync gets called

. Any ideas? And this time I promise that I have completely read and understood man 5 make.conf and has not seen such an option.
_________________
There are 10 kinds of people in the world,
those who understand binary and who don't

yetano · n00b Joined: 02 Mar 2005 Posts: 2

jkroon · Posted: Wed Mar 23, 2005 5:35 am Post subject:

I think I'll just go with the additional rsync command in our nightly scripts - it is imho the cleaner solution. Also make it easier to distinguish which is custom ebuilds and which are portage ebuilds. It doesn't solves the problem for all but a small number of desktops, which isn't that big a problem - we can probably just write a small script that does both emerge sync and rsync -rav --delete rsync://out.rsync.server/gentoo-local-portage /usr/local/portage in one go.
_________________
There are 10 kinds of people in the world,
those who understand binary and who don't

jkroon · Posted: Mon Mar 28, 2005 3:18 pm Post subject:

Ok, torpage 0.2.0 is available for testing. It still lacks ACLs but who cares? For most of us this is simply a caching solution and EXCLUDEFROM can be used to exclude games-* (which is what I wanted to have ACLs for).

The new version instructs the server to fetch a specific file for a specific package and version, the server then manually parses the appropriate digest files to check whether these files are actually in those packages, and then does some nifty parsing and sourcing of files to determine the upstream mirrors that portage would have fetched from and then manually fetches and does the checksumming. I do this by hand since portage cannot fetch a particular file for me at this point in time - nor is the emerge command available on non-gentoo servers (and yes, I do have the need that torpage run on non-gentoo systems on the server side).

Additionally torpage doesn't take over your fetching entirely any more, it now actually specifies a seperate torpage:// protocol, so you can have multiple upstream torpage servers (ok, that is just leetness factor and doesn't imho have much real use) along with other standard ftp:// and http:// servers and torpage won't interfere with these. This allows for fallback should torpage fail for some reason or another. I've even managed to configure it to use different upstream torpage servers depending on the network I'm connected to, or to silently fail if I'm not connected to one of these networks.

Anyway, please take a look and let me know of any shortcommings (both in my code and in documentation).

URI: http://www.kroon.co.za/torpage.php

Thanks for any and all feedback.
_________________
There are 10 kinds of people in the world,
those who understand binary and who don't

mndar · n00b Joined: 04 Apr 2005 Posts: 52

HI,
Is there any way I could get back my original emerge cache/ portage tree, after doing emerge --sync
Sorry, if this question is too silly to be replied to, by all the GURUs out there. I recently installed Gentoo from the universal CD (install-x86-universal-2005.0.iso) and then installed KDE, GNOME, etc from the package CD (packages-x86-2005.0.iso) using emerge --usepkg <package-name> This installed KDE 3.3.2.
Well, then I did emerge --sync. Now I can't install any kde-3.3.2 package since all the entries in the emerge cache/portage tree point to kde-3.4. Thus I can't use the package CD. I want to be able to install packages from the Package CD which has got kde-3.3.2. So, is there a way by which I could get back my original emerge cache or portage tree. I was thinking emerge --regen. Would that do it ?
Also I am not very sure whats the difference (if any) between emerge cache and portage tree.

jkroon · Posted: Tue Apr 05, 2005 6:29 am Post subject:

You presumably got your orriginal portage tree from a tarball on the CD? I suspect the following sequence of commands will solve you problem in that case:

mndar · n00b Joined: 04 Apr 2005 Posts: 52

Thanks jkroon, it worked!!
Following were the sequence of commands I had to issue

yaneurabeya · Veteran Joined: 13 May 2004 Posts: 1754 Location: Seattle

If you have several machines as well with different configurations, and several flagship machines that do compiling, etc... then you might want to consider something like this...

Server
/ \
/ \
/ \
PC(0) [....] PC(n)

Where PC(0) through PC(n) have NFS access and are trusted for authentication. Given that each and every machine can mount and access the distfiles NFS share, you can setup a simple cron command where the trusted machines will 'sync' with the server by running cp /usr/portage/distfiles/*. Then you can run rsync to your heart's content. Why not just use NFS? Because NFS requires an additional service for all the non-flagship machines and rsync doesn't require any additional kernel modules.

jasperbg · Last edited by jasperbg on Tue Jun 07, 2005 6:00 am; edited 1 time in total

jkroon · Posted: Tue Jun 07, 2005 5:55 am Post subject:

You really don't want to be doing any kind of NFS unless your _entire_ network can be trusted. It is simply too insecure. Take for example a university (Yes, I'm a sysad at a university) where I'll need to export /usr/portage rw to the entire campus (well, a large part thereof anyway) to make your trick work. Not something I'm willing to do since this means some cracker can go and put a new baselayout along with a trojanned archive on the server and everybody would install it.

Nope, not a good idea imho, plus there is locking issues even if you do trust the entire network.
_________________
There are 10 kinds of people in the world,
those who understand binary and who don't

yaneurabeya · Veteran Joined: 13 May 2004 Posts: 1754 Location: Seattle

Yes, and that's why something should be done for authentication dealing with trusted keys and so forth like kerberos auth.

jkroon · Posted: Thu Jun 09, 2005 5:41 am Post subject:

matt.matolcsi · n00b Joined: 03 Apr 2005 Posts: 5

First off, thanks to everyone for the well-written, thoughtful, and useful comments.

Here at work we've got something of a similar situation: five recycled Ultra 5's do most of the dirty server work, with one of them being the designated binary package build machine and portage sync'er. Because they are all very similar hardware (only processor speed and RAM differ), they all use the same make.conf and general filesystem layout. Actually, I've got a neat rsync setup that lets the other machines pull this and some other files automatically from the server.
At first, I setup the machines to use binary package downloading using emerge -g, but this had some drawbacks:

1. It's slow: every time I wanted to emerge a package, some 'metadata pickle' had to be downloaded and parsed, and this took ages.
2. It required the setup of an http server on the package host, which was one more thing to worry about that I didn't want.

I also had that host serve as the rsync mirror, which had another few disadvantages:

1. It came off of an actual hard drive, which meant that it was probably 5-10x slower than syncing to an official mirror (which act like they're on RAM drives.)
2. It caused a ton of grinding each time the tree was sync'd! This worries me because I want these servers to go the next 5 years with minimal adjustments on the administrator's part (I'll be leaving soon), and all that grinding * 4 for each machine that had to sync just gave me a bad feeling.

Solution? Share /usr/portage read-only over NFS to the local network. The other machines automount /usr/portage, and I can use emerge --buildpkg (actually, I have it set in FEATURES in /etc/make.conf to build binary packages) to build packages on the main host, and then install them on the others using emerge --usepkg.

Actually installing packages has worked great, but I'm worried because emerge goes through a huge round of 'quarterly updates' when I first automounted /usr/portage, and sometimes it acts like it wants to write into those directories:

Performing Global Updates: /usr/portage/profiles/updates/4Q-2004
(Could take a couple of minutes if you have a lot of binary packages.)
.='update pass' *='binary update' @='/var/db move'
s='/var/db SLOT move' S='binary SLOT move' p='update /etc/portage/package.*'
......................................!!! Cannot update readonly binary: sys-devel/gcc-3.3.5-r1
!!! Cannot update readonly binary: sys-devel/gcc-3.3.5.20050130-r1
.....!!! Cannot update readonly binary: sys-devel/libtool-1.5.2-r5
..........................................................................................................................................................................................................................

Does anyone know if indeed anything *needs* write access within /usr/portage? It seems like nothing should, since all the ebuilds get synced, but it still worries me that emerge complains.

Thanks,

Matt

yaneurabeya · Veteran Joined: 13 May 2004 Posts: 1754 Location: Seattle

Uhm, you need +w for distfiles and certain sections of portage (individual branches, ie patches) I think.

matt.matolcsi · n00b Joined: 03 Apr 2005 Posts: 5

distfiles I'm not worried about, since only one machine will sync with the outside world and download/build packages; but what process or program needs to be able to write to specific ebuilds?

jkroon · Posted: Sun Jul 03, 2005 11:31 am Post subject:

I've submitted a patch that removes the need for +w to ${DISTDIR}, but then you need some other method to fetch files (like torpage - http://www.kroon.co.za/torpage.php works nicely). I like the --usepkg and --buildpkg idea, I quess one can even use the same idea that torpage uses for fetching distfiles for building packages ... (aka, packages gets built on demand whilst the client waits).
_________________
There are 10 kinds of people in the world,
those who understand binary and who don't

bigdave1 · Posted: Tue Jan 10, 2006 2:02 am Post subject:

I am having a problem setting this up as described in Setup #3. I have set up rsync exactly as described and setup my client accordingly. However, whenever I do an emerge --sync, I get the following error:

jkroon · Posted: Tue Jan 10, 2006 6:40 pm Post subject:

From the client machine, what does the command:

bigdave1 · Posted: Thu Jan 12, 2006 12:09 am Post subject:

Well, I figured out my problem, I was making changes to the /etc/rsync/rsyncd.conf file instead of the /etc/rsyncd.conf file. However, I now have a new problem. What I would like to happen when I emerge something (ie. emerge gdesklets-core), have my portage server download the ebuild for that package all and the dependencies if its not there and cache it in its portage directory, and then serve it to the client that requested it. However, thats not what is happening. When I do a 'emerge gdesklets-core' here's what I get:

jkroon · Posted: Thu Jan 12, 2006 2:20 am Post subject:

bigdave1 · Posted: Thu Jan 12, 2006 12:54 pm Post subject:

Cool.. Now have an even better question. Now that I've downloaded the ebuild for torpage, where do I save it and how do I get it to emerge?

Thanks!

jkroon · Posted: Thu Jan 12, 2006 1:50 pm Post subject:

bigdave1 · Posted: Fri Jan 13, 2006 2:10 am Post subject:

Ok, this may sound like a stupid question, but do I need to install this on the server or the client or both? According to the readme on the link you provided, it sounds like both. I just want to make sure so I get this right.

Thanks!

bigdave1 · Posted: Fri Jan 13, 2006 2:51 am Post subject:

Ok, well I installed torpage onto the server and client anyway. So, disregard my previous post. However, when I try to emerge something it doesn't pull it from my portage server. And my portage server doesn't go out and retrieve it. Here's what I get when I try to emerge gdesklets-core: