| View previous topic :: View next topic |
| Author |
Message |
GurliGebis
Retired Dev
Joined: 08 Aug 2002
Posts: 509
|
 Posted: Tue Jun 17, 2003 6:11 pm Post subject: |
 |
|
Currently I only have 4 machines behind the server/router.
There is no problems with accessing the files, since I let one machine download the files, and then install it on the other machine after it is done.
I think (99.99% sure) having several machines writing to the same file would corrupt it.
_________________
Queen Rocks. |
|
| Back to top |
|
 |
GurliGebis
Retired Dev
Joined: 08 Aug 2002
Posts: 509
|
| Posted: Tue Jun 17, 2003 6:15 pm Post subject: |
|
|
And about the rsync problem, I have the server doing it at midnight, and the clients doing it ever 4 hour. (Not hitting midnight) 
_________________
Queen Rocks. |
|
| Back to top |
|
|
hackertype
n00b
Joined: 03 Jun 2003
Posts: 32
|
| Posted: Wed Jun 18, 2003 2:43 am Post subject: |
|
|
When I emerge on a client machine it would be nice if the client would rsync the binary from my rsync mirror. Most of my machines are the same arch. and binaries are compatible.
Will changing the PKGDIR to point to the rsync mirror work? If not I suppose I could rsync the packages dir on a regular basis.
Any idears? |
|
| Back to top |
|
|
swtaylor
Retired Dev
Joined: 18 Jun 2003
Posts: 2
Location: Loveland, CO
|
| Posted: Wed Jun 18, 2003 7:02 am Post subject: Recommendation: add a partial source mirror in your network |
|
|
Within my network, i've got the portage rsync mirror (to which a script also pulls down the latest gnome ebuilds from breakmygentoo.net) and applies to my tree because i like to live on the edge 
Since the sources that my machines download tend to have a lot in common, i fired up a vsftpd instance on my rsync server to allow anonymous read-only access to the files within my /usr/portage/distfiles, and then i add it by hand as the first server in the GENTOO_MIRRORS list of all the other machines on my network. This way, any packages this server has already acquired is immediately available at full LAN speed to the rest of my network. Packages not on there just failover to the typical search of mirrors on the net.
The typical day-to-day emerge -U on that system keeps the distfiles fairly current, and i'll execute an emerge --fetchonly on it to pre-cache builds which i don't necessarily want installed on it, but want available for my other machines. |
|
| Back to top |
|
|
Macros73
n00b
Joined: 04 Jun 2002
Posts: 25
|
| Posted: Wed Jun 18, 2003 12:20 pm Post subject: |
|
|
The only problem with using rsync, unless I'm grossly misunderstanding something, is that emerged distfiles will still take up space on the local client.
On our small network, I have one server configured as an rsync master for 'emerge sync', and as an NFS master for sharing the /usr/portage/distfiles directory. All clients rw access to distfiles, which I admit is an insecure nightmare waiting to happen. All clients also have access to the Internet for wget.
This has a couple beautiful effects, however:
1. The client systems don't store packages locally. When they emerge a package, it uncompresses from the NFS share.
2. If the requested package doesn't exist on the NFS share, it is automatically downloaded and added to the NFS share by the client.
The Ideal Solution would be a Portage Proxy that would work something like this:
1. Client requests package X from server.
2. Server downloads package X automatically if it is not already available.
3. Client install the package, deletes work and package files from local system afterwards.
4. (Optional) Client notifies server package installed for accounting purposes. |
|
| Back to top |
|
|
TheQuickBrownFox
n00b
Joined: 08 Oct 2002
Posts: 37
|
| Posted: Thu Jun 19, 2003 9:04 am Post subject: |
|
|
I'm converting some labs to gentoo, but I'm still trying to figure out how to automagically deal with config files. 
The setup:
One rsync server for the whole campus, also mirroring all the distfiles. (rsync + ftp)
Two NFS/NIS servers. One for undergrads, one for postgrads, each serving different users in different labs.
Different labs have different hardware, but all the hardware in a lab is identical.
So what I'm doing is this:
1. On each server, create a directory, say /home/gentoo (I used /home/gentoo, because /home is where I have space) to have a chrooted lab "image".
2. Then install the whole lab image (except maybe the kernel) with the USE flags specific to this lab configuration in this directory, using "emerge -b" so that binary packages are built and installed.
3. Export /home/gentoo/usr/portage (ro) Run "emerge sync && emerge -b world && etc-update" in this chrooted environment when neccessary to update the clients.
4. On the client machines, just mount server:/home/gentoo/usr/portage on /usr/portage. Then all the client needs to do, is "emerge -k world" to sync.
Keeping a whole chrooted distro around may sound like overkill, but when the time comes that you want to quickly install a client machine with a known good configuration, you will be thankfull. Using a previous client is not feasible, because you can't garauntee that it wasn't messed up (or rooted) in some obscure way by someone.
There is one more problem: How do I deal with config files after upgrading packages? Would it be feasible to have another set of ebuilds, like (system, world, local) so that I could "emerge -u local" after "emerge -u world" to fix the configs?
_________________
-- jumps over the lazy dog |
|
| Back to top |
|
|
thinair
Tux's lil' helper
Joined: 01 Nov 2002
Posts: 144
Location: Suisse
|
| Posted: Thu Jun 19, 2003 12:23 pm Post subject: |
|
|
I this tip seem to be very good.. I haven't try it for the moment, but I will do it next week.
I would like to know, if you allow that I translat it in french as a tutoriel (howto) for
http://doc.gentoofr.org and probably I will some usefull part.
Naturally I would notice that you are the author of this post (if you want), but to do that I need that you PM me your name (or just let your nick) and an e-mail for contact purpose (in case it's neccessary).
thanx,
thinair
_________________
http://www.gentoofr.org
Le site francophone de GNU/Linux Gentoo |
|
| Back to top |
|
|
Koon
Retired Dev
Joined: 10 Dec 2002
Posts: 518
|
| Posted: Thu Jun 19, 2003 3:26 pm Post subject: |
|
|
So many different setups... all very good for their purposes. It might be difficult to promote some reference setups since there are many many options.
Thinair : before translating, you might prefer to wait for a more comprehensive version of the HOWTO (one which would include description of other setups, like the NFS tree sharing or the proxy/cache of the distfiles...).
We're still in the process of collecting information on people setups, then we'll have to choose what to promote in the HOWTO (that will be the difficult part) then share the work of writing parts of the HOWTO, everything being merged by Grimthorn, since he is the first poster...
So : if you use a different setup to achieve the same purpose, please post your architecture here !
-K |
|
| Back to top |
|
|
thinair
Tux's lil' helper
Joined: 01 Nov 2002
Posts: 144
Location: Suisse
|
| Posted: Thu Jun 19, 2003 3:50 pm Post subject: |
|
|
I am not currently using centralized rsync... but I am planning to use it for my small network.
If I found other solution or a more complete way to answer to this problem. I will post it here too.
++
thinair
_________________
http://www.gentoofr.org
Le site francophone de GNU/Linux Gentoo |
|
| Back to top |
|
|
Yak
Tux's lil' helper
Joined: 01 Sep 2002
Posts: 107
|
| Posted: Fri Jun 20, 2003 7:07 am Post subject: |
|
|
My solution for packages was to setup proftpd on my 'portage gateway' (it's behind a firewall) and put the distfiles directory under the ftp directory and allow anon ftp access from local network. Not too elegant, but was quick to setup and still works like a charm. This way, all there is to setting up a client is adding your portage gateway first in GENTOO_MIRRORS as ftp://yourgateway
I guess this would have the advantage of being quick and easy to type in clients' /etc/make.conf, and fetches files automatically if they are not on your server. Only problem is then that new package is on the client and not the server.. o well there's always nfs and cp -u to get them back to the server 
(I can laugh b/c I only mess with 2 or 3 not 20+ gentoos) But these kinds of tips saved me DAYS of downloading at 24Kbps  Good to see it all wrote up so nice in one place. |
|
| Back to top |
|
|
savage
Apprentice
Joined: 01 Jan 2003
Posts: 161
|
| Posted: Sat Jul 05, 2003 2:22 am Post subject: Re: HOWTO: Central Gentoo Mirror for your Internal Network |
|
|
Update ! see end of page 3 for a better solution (and one that works with the latest Gentoo!)
Great work Grimthorn!
Here is my modification that works well (automatically downloads files as they are requested if they are not already on my gateway server).
Uses apache with php4 and a simple C program.
Here is the php4 script: (just stuck in /home/httpd/htdocs/getFile.php on the server)
--start getFile.php--
| Code: |
<?
$packageSrc = urldecode($_GET["packageSrc"]);
$packageName = urldecode($_GET["packageName"]);
if($packageName != "")
{
@$fileHandle = fopen("/usr/portage/distfiles/" . $packageName, "r");
if(!$fileHandle)
{
exec("/usr/local/sbin/getPackageFromMirror $packageSrc $packageName");
@$fileHandle = fopen("/usr/portage/distfiles/" . $packageName, "r");
if(!$fileHandle)
{
print "Unable to get File from remote Server\n";
}
else
{
fpassthru($fileHandle);
}
}
else
{
fpassthru($fileHandle);
}
}
?>
|
--end getFile.php
Here is the c program. Compile it with:
gcc -s -o getPackageFromMirror getPackageFromMirror.c
cp getPackageFromMirror /usr/local/sbin/getPackageFromMirror
chown root.root /usr/local/sbin/getPackageFromMirror
chmod 4755 /usr/local/sbin/getPackageFromMirror
^^^^^^^^^
Yes, I know that has security implications, but if you are savy enough to see that, you are probably savy enough to see it is only a problem if you have other users on your gateway box. If this is your setup, I suggest sudo or something like that. Basically, the binary needs to be owned by whoever can write a file to your /usr/portage/distfiles directory on your server, and needs to be suid that user.
--start getPackageFromMirror.c
| Code: |
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <strings.h>
#include <errno.h>
int main(int argc, char* argv[])
{
extern int errno;
int i=0;
char wgetCommand[1024];
char fileTarget[1024];
if (argc != 3)
{
printf("Usage: %s <packageToRetrieve> <packageName>\n", argv[0]);
return(-1);
}
bzero(wgetCommand,1024);
bzero(fileTarget,1024);
snprintf(wgetCommand,1024,argv[1]);
snprintf(fileTarget,1024,"/usr/portage/distfiles/%s",argv[2]);
printf("/usr/bin/wget -q -N -O \"%s\" \"%s\"\n", fileTarget,wgetCommand);
execl("/usr/bin/wget","wget","-q","-N","-O",fileTarget,wgetCommand);
printf("%s\n",strerror(errno));
}
|
--end getPackageFromMirror.c
///end server setup.
This setup required few modifications to make.conf (on the clients). Go ahead and setup / leave your setup however with you GENTOO_MIRRORS setting - this will tell the server where to try to get the file from.
The only line you need to modify is the FETCHCOMMAND. Observer the one I use:
| Code: |
FETCHCOMMAND='/usr/bin/wget -t 5 http://[proxy]/getFile.php?packageSrc=\${URI}\&packageName=\${FILE} -O \${DISTDIR}/\${FILE}'
|
With this setup, it is possible to use the downloads on your proxy box (no need for a seperate cache); also, any files requested by any clients that are not on the local machine are downloaded from the appropriate source.
Hope someone out there finds this cool and useful.
Last edited by savage on Wed Feb 04, 2004 3:27 am; edited 3 times in total |
|
| Back to top |
|
|
neptune
n00b
Joined: 11 Oct 2002
Posts: 49
|
| Posted: Tue Jul 08, 2003 7:06 am Post subject: just nfs |
|
|
Hello, nice trick, but I'll try to do simpler.
One box that shares /usr/portage and the others that mount it (eg: /mnt/portage). on the others, modifiy /etc/make.conf to have PORTDIR=/mnt/portage and DISTFILES=${PORTDIR}/distfiles
The box that shares do a rsync every day. the others doesn't need to do it. If one box download a source, the others doesn't need to download it again.
what do you think of it? |
|
| Back to top |
|
|
paulwoodwarduk
n00b
Joined: 11 Jul 2003
Posts: 1
Location: UK
|
| Posted: Fri Jul 11, 2003 7:34 am Post subject: error in File listing 2.3 |
|
|
had to adjust the custom fetchcommand slightly - it'll catch out other "cut 'n paste" monkeys too!
FETCHCOMMAND=rsync rsync://<your Portage gateways IP or DNS>/gentoo-packages/\${FILE} {DISTDIR}
should be:
FETCHCOMMAND=rsync rsync://<your Portage gateways IP or DNS>/gentoo-packages/\${FILE} ${DISTDIR}
had me going for 15 mins! Is it possible for the original "HOWTO" to be edited, especially with the mod required to /etc/init.d/rsync to remove compression etc? |
|
| Back to top |
|
|
icarus1983
n00b
Joined: 12 Jul 2003
Posts: 27
Location: Norway
|
| Posted: Sun Jul 13, 2003 9:47 pm Post subject: |
|
|
| Thorbjorn wrote: | here is my mod_proxy with caching support.
tis is the first time i setup mod_proxy in apache, but it is working fine for me. I need to tweek with the cache settings a bit maybee but I would exspect you all would need to. One more thing i want to do is get the ftp proxy working. My initial attempots at gettin that goin were not succsefful. I get a segfault on child processes when trying to proxy ftp.. I am not sure if this is due to the way the mod_proxy was compiled or not, and not alot of info on google about it. Anyone know whats goin on there ?
My approach was to setup an internal caching proxy on a virtual host as not to interupt anything else runnign on this box. I assum you know how to uncoomment the LoadModule directives in your apache configs ( mod_proxy is a std module) and I assume you are using apache 1.3.x ( for 2.0 your config would be a bit different). I also assume you know how to setup a virt host, but you can slap this in a location directive if you want to.
Here is my apache virtual host:
| Code: |
<VirtualHost yourwhatever >
# we dont want to proxy just do ProxyPass This is a security measure:
ProxyRequests Off
# where the cache files live ( apache needs to be albe to write here )
CacheRoot "/somedir/cache/httpd"
CacheSize 102400
CacheGcInterval 8
CacheMaxExpire 168
CacheLastModifiedFactor 0.1
CacheDefaultExpire 72
#setup the site we want to proxy
ProxyPass /gentoo/ http://csociety-ftp.ecn.purdue.edu/pub/gentoo/
# setup some restrictions on who can connect
<Directory proxy:*>
Order deny,allow
Deny from all
Allow from <your domain>, <yourip>
</Directory>
</VirtualHost>
|
and Viola any request to yourvirt/gentoo/ is transparently proxied, and all downloads from said URL are cached on your local server...
Each of the seven Cache directives, controls how the server handles caching. Setting the CacheRoot enables caching on the server. This directory must be writable by the user running the server (usually "nobody"). The CacheSize sets the desired space usage in kilobytes. You will probably want to set this higher than the default of 5, based on your available disk space, to allow the greatest number of documents to be stored locally, thus allowing local cache access by the clients. Garbage collection, which enforces the cache size, is set in hours by the CacheGcInterval. If unspecified, the cache size will grow until disk space runs out. CacheMaxExpire specifies the maximum number of hours for which cached documents will be retained without checking the host server. If the origin server for a document did not send an expiry date, then the CacheLastModifiedFactor will be used to estimate one by multiplying the factor by the time the document was last modified. If the protocol used to retrieve a document does not support expiry times (FTP, for example), the CacheDefaultExpire directive specifies the number of hours until it expires.
now just edit your make.conf to point to yourserver like so:
| Code: |
GENTOO_MIRRORS="http://myproxy/gentoo"
|
and all is good. sit back and enjoy not sucking up all your bandwidth building new gcc on emerge -u world on the 12 cluster nodes in your garage.. er.. or whatever you may have
Edit Added the info about the cache vars. |
Yes, that's the best solution so far... I thought... Untill I tried it.
After putting it all together I tested it by emergeing a plugin for gkrellm, gkrellmoon.
It worked, and I browsed my cache server to look for the file and when I found it, it looked like this.
| Code: | | 60K -rw------- 1 apache apache 54K Jul 14 23:32 /home/cache/httpd/2/r/X/utNJZyzeNb@jiS7HhJw |
the file on the client looks like this:
| Code: | | 60K -rw-r--r-- 1 root root 54K Mar 24 17:11 gkrellmoon-0.6.tar.gz |
Maybe it's me, but if I'm going to be caching portage I'd like real filenames! I guess it's not possible to i.e backup this kind of cache? Is there an option in the apache config to make it cache real names ?
_________________
Where did you want to go today did you say ? |
|
| Back to top |
|
|
r_yuan
n00b
Joined: 07 Jun 2003
Posts: 66
Location: Montreal, Canada
|
| Posted: Sat Jul 26, 2003 6:56 pm Post subject: |
|
|
| cwng wrote: | [...] I prefered a http method in GENTOO_MIRRORS so that if a source tar is not in the gateway, emerge will failover and use an alternative mirror.
To that effect, I emerged 'mini_httpd' (apache is an overkill, unless you actually already have apache set up). |
I have researched a python-based web server. The scripting should be fairly simple. I piggybacked the portage script so that it follows the settings in /etc/make.conf.
| Code: | #!/usr/bin/env python2.2
import SimpleHTTPServer
import portage
import os.path
BaseClass = SimpleHTTPServer.SimpleHTTPRequestHandler
class MyRequestHandler(BaseClass):
def send_head(self):
if self.path.startswith('/gentoo') and (len(self.path) > len('/gentoo/')):
myfile=os.path.basename(self.path)
try:
return open(portage.settings["DISTDIR"]+"/"+myfile,'r')
except IOError:
# beware of threaded server -- unverified
# parallel fetch'es _may_ cause db corruption
if portage.fetch([myfile], listonly=0, fetchonly=1):
self.send_response(200)
self.send_header("Content-type", "text/plain")
self.end_headers()
return open(portage.settings["DISTDIR"]+"/"+myfile,'r')
else:
return BaseClass.send_head(self)
else:
return BaseClass.send_head(self)
def test():
SimpleHTTPServer.test(MyRequestHandler)
test()
## Scripting the Web with Python
# http://www.w3j.com/6/s3.vanrossum.html
# http://python.org/doc/current/lib/module-SimpleHTTPServer.html
# /usr/lib/python2.2/site-packages/portage.py
# /usr/lib/portage/bin/emerge
# /usr/portage/w3j2.py # _this_ file [ server]
# /etc/make.conf # GENTOO_MIRRORS="http://myproxy:8000/gentoo" [ client ]
# http://forums.gentoo.org/viewtopic.php?t=59134
|
On the server, copy the script into /usr/portage/w3j.py. It should be owned by root and executable. Start the server with 'w3j2.py 80' (for port 80, otherwise it defaults to port 8000). On the clients, insert 'GENTOO_MIRRORS="http://myproxy/gentoo"' in /etc/make.conf.
Caveats:
1. This is quick solution (for those who don't want to install Apache). The performance of this server isn't great.
2. I have not verfied whether or not parallel fetches will cause database corruptions -- please report back to me. I'm not even sure whether or not the server is threaded in any way.
3. Fetches are not streamed ie: the file download must be completed before it is forwarded to the client. I hope transparent proxies work better than this. |
|
| Back to top |
|
|
0problem.dk
n00b
Joined: 13 Jul 2003
Posts: 5
Location: Ikast, Denmark
|
| Posted: Thu Aug 14, 2003 10:39 am Post subject: |
|
|
| Thorbjorn wrote: | ...
Here is my apache virtual host:
| Code: |
<VirtualHost yourwhatever >
# we dont want to proxy just do ProxyPass This is a security measure:
ProxyRequests Off
# where the cache files live ( apache needs to be albe to write here )
CacheRoot "/somedir/cache/httpd"
CacheSize 102400
CacheGcInterval 8
CacheMaxExpire 168
CacheLastModifiedFactor 0.1
CacheDefaultExpire 72
#setup the site we want to proxy
ProxyPass /gentoo/ http://csociety-ftp.ecn.purdue.edu/pub/gentoo/
# setup some restrictions on who can connect
<Directory proxy:*>
Order deny,allow
Deny from all
Allow from <your domain>, <yourip>
</Directory>
</VirtualHost>
|
...
|
As you said, it is a bit different on Apache 2.
The above will give you the files you request, but it will not do the caching, which was the primary purpose of the setup.
However, you only need one more line to make it work as intended:
| Code: | | CacheEnable disk /gentoo/ |
|
|
| Back to top |
|
|
Lightspeed
n00b
Joined: 21 Feb 2003
Posts: 57
Location: UK
|
| Posted: Sun Nov 16, 2003 9:35 pm Post subject: |
|
|
I wanted to setup something similar on my network, but the only machine I have which is permanently running is a Windows Server 2003 box (although it does at least have cygwin installed). So I came up with a solution that works for a Windows machine:
Considering the uncertain current state of the gentoo-cygwin project, I decided to manually schedule the server to download the latest portage tree snapshot from my nearest gentoo mirror every day and unpack it. The portage tree is then shared as read only and mounted by my gentoo machines over samba to /usr/portage. Before the server attempts to update the portage tree it will first make a copy of it, then unshare the main tree directory and share the copy instead. The updated tree is placed in the main directory and the share is moved back to that folder, then the copy is deleted.
For the distfiles, I have a central distfile folder which is served up using Apache (I had originally tried using IIS 6 simply because that is what the rest of my intranet site uses, but for some reason that I still cannot understand all .tar.bz2 files were giving me 404 errors, when others like .tar.gz were downloading fine). The gentoo machines point to this webserver as their primary mirror in /etc/make.conf. The server also has a separate rw shared directory for each gentoo machine, which the gentoo boxes mount using samba into /usr/portage/distfiles. So the distfiles that a gentoo machine downloads, whether from the LAN server or the internet, are still kept on the server, and they don't interfere with other machines emerging at the same time. Every day the server will then check to see if the gentoo machines' personal distfile directories contain anything that the main central distfile folder doesn't have, and if so it copies them across.
You may think that this method of having a separate distfile share on the server for each gentoo box is ridiculously wasteful in space, but I am running the SIS (Single Instance Storage) groveler service that is installed with Windows' RIS (Remote Installation Services) software which will search the distfile directories for identical files, and replace them all with NTFS junction points linking to a single copy of the duplicated file. |
|
| Back to top |
|
|
Helena
Veteran
Joined: 02 Apr 2003
Posts: 1114
Location: Den Dolder, The Netherlands
|
| Posted: Mon Nov 17, 2003 1:40 pm Post subject: |
|
|
Just a simple, perhaps basic question. What rsync command is the best?
Let me explain. I've adopted the rsync solution for several reasons. So I have one Gentoo box acting as a "central Gentoo mirror" as Grimthorn calls it (BTW thx for the guide, that's just what I needed). This one has IP address 10.0.0.200 (i'm exploring LAN name resolution, so I must use IP addresses for the moment).
I have set up a separate partition for holding all the Gentoo distribution files, although that is not required; I just find it convenient from an administrator's viewpoint. Since all I want is to distribute updates to local clients (installation is done from LiveCDs) I only mirror "distfiles" and "snapshots" from the source mirror, as suggested by the official guide http://www.gentoo.org/doc/en/source_mirrors.xml.
I also chose to set this mirror up both as an rsync- and FTP-server (not HTTP). For the rsync part I follow Grimthorn's guidelines, for FTP I've setup vsftpd and xinetd. So I've now edited all client's /etc/make.conf to contain the following extra lines:
| Code: | SYNC="rsync://10.0.0.200/gentoo-portage"
GENTOO_MIRRORS="ftp://10.0.0.200/gentoo"
|
For the server itself, I use a slghtly different form, as a prelude to name resolution which I still want to implement om my LAN later (I've already read some threads about this, but any help would still be welcome...)
| Code: | SYNC="rsync://localhost/gentoo-portage"
GENTOO_MIRRORS="ftp://localhost/gentoo"
|
After some experimentation I now use the following rync commands, where /mnt/rsync/gentoorsyncmirror/ points to my local mirror:
for the packages, following the official guide,for the Portage tree (since ibiblio.org does not mirror the Portage tree I chose the closest mirror), andfor the Portage tree snapshots, although I'm not sure why I would need them. Any suggestions for improvement?
Sorry about using the long form of command switches but since it's automated I prefer that. |
|
| Back to top |
|
|
brendaniabbatis
n00b
Joined: 25 Nov 2003
Posts: 54
Location: Sacramento, CA, USA
|
| Posted: Wed Nov 26, 2003 1:00 am Post subject: Gateway under limited Internet connection |
|
|
This HowTo has some great ideas. My plan has an interesting twist I thought I would share:
I intend to use my laptop for a Gentoo gateway on my home network, to take advantage of the high bandwidth at work to bring the updates home. I have no Internet access at home, an easy security solution.
The gateway is the only system connected to the Internet, and only at certain times. It will update the portage tree by rsync each time it is connected, but a maximum of once per day.
Clients access the portage tree and distfiles by read-only nfs mount. Distfile fetches are directed to the proxy, but always fail. The gateway then logs the request and retrieves the needed files the next time it connects to the Internet, and the clients can try again later.
The portage tree is only available when the laptop is home, but I am the only one who maintains the systems so it does not matter. Alternatively, the tree and distfiles could be copied to clients using rsync, or didn't I see somewhere once an automatic synchronizing filesystem?
The drawback to this is that updates are not immediately available, but so far my experience with Gentoo is that compiling two or three packages a day is as much as I want to do, leaving the rest for tomorrow.
If I am not mistaken, distfiles are unpacked in a directory in /usr/portage, so that has to be set to something else in make.conf (?). Seems to me that is true anytime nfs is used for the portage tree.
I had a thought as I read this. Perhaps once a consolidated HowTo of all the good ideas here is done, an ebuild could also be written for each of the various solutions. Maybe that's a real n00b idea. |
|
| Back to top |
|
|
bendy
Apprentice
Joined: 18 May 2002
Posts: 274
Location: Gloucester, UK
|
| Posted: Sun Nov 30, 2003 3:40 pm Post subject: |
|
|
Hi,
I've been trying to set up a "half and half" solution i.e. I set up an rsync mirror on my lan for the portage tree only, and then share /usr/portage via nfs from the same machine.
If I mount /usr/portage on the client, then run emerge sync, I get the following: | Code: |
root@laptop root # emerge sync
>>> starting rsync with rsync://newserver/gentoo-portage...
This is rsync[number].[country].gentoo.org.
receiving file list ...
70284 files to consider
io timeout after 180 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(103)
rsync: connection unexpectedly closed (1655051 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165) |
However, if I run emerge sync with /usr/portage unmounted then the sync works but I get all the usual directories written locally to the (unmounted) /usr/portage.
Is this normal behaviour?
bendy. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Tue Dec 02, 2003 3:24 pm Post subject: Another Option - tsp-cache...... |
|
|
Setting up your own rsync works the gentoo way for emerge syc. But what about emerge -u world???
Well, I've been looking for a way to speed up my system. I have a fast lan but still had to waste bandwidth downloading files multiple times. All the options listed in this thread seem to have some serious drawbacks. Since I couldn't find what I wanted - I tried to make a better option.
Check out tsp-cache
https://forums.gentoo.org/viewtopic.php?t=110973
tsp-cache... It is a cache designed for gentoo that works with no changes to client systems... only adding a mirror to make.conf
Advantages:
1. Bandwidth savings. The file is only downloaded once no matter how many clients request the file.
2. No Portage problems. tsp-cache is transparent to clients. tsp-cache is just listed as one of the many download mirrors in make.conf. Failure of the tsp-cache host machine will not cripple the clients. They will automatically use the next available mirror like normal.
3. Streaming to clients. tsp-cache will download the file from a mirror while simultaneously streaming the file to multiple clients. Therefore, client emerges will not fail even though the files are not in the cache. I can even emerge -u world on multiple machines and all file updates only get downloaded once - I don't even have to wait for the first machine to finish downloading the updates before starting the update on my other machines!!!
4. [b]Reports.[/] tsp-cache can create html reports of cache efficiency and bandwidth.
I've got a full HOWTO in the thread https://forums.gentoo.org/viewtopic.php?t=110973 |
|
| Back to top |
|
|
tiny
Tux's lil' helper
Joined: 02 Dec 2003
Posts: 102
Location: Slovenia
|
| Posted: Tue Dec 02, 2003 4:48 pm Post subject: |
|
|
Hi!
I'm a gentoo noob so plz go easy on me. 
Otherwise I have 6-7 years of experience with Linux.
I recently installed Gentoo at my home and I liked it immediately.
Just the thing I was searching for in Linux.
Ok to cut things short. I'd like to install Gentoo at my work to.
I'm in control of 3 machines all running Linux. One of them could be
my Gentoo gateway, since bandwidth over the day is very poor and
downloading would require a lot of time.
But this machine isn't running Gentoo distribution and it needs to stay this
way (for now). The only thing Im looking for is fast network access to
ebuild packages. Portage can be on the machine I'm installing Gentoo to.
Question is how can I keep packages on my server machine (gateway)
up to date?
Lets say I put them in apache directories. What is the simplest method of
keeping those packages up to date on regular bases?
I was thinking in this way ... simple bash script with wget would do it.
I decide which packages I wont, I DL them and later on I keep
them updated. Only how would I keep them updated so I dont burn a lot of
bandwith. Is there posibilty to use rsync from my Linux distribution?
Or something like that.
Storage space isn't problem.
Oh BTW, how much space would take if I DL all the packages. I'm not
gonna, I'm just wondering.
regards,
Tiny |
|
| Back to top |
|
|
Helena
Veteran
Joined: 02 Apr 2003
Posts: 1114
Location: Den Dolder, The Netherlands
|
| Posted: Tue Dec 02, 2003 7:59 pm Post subject: |
|
|
| tiny wrote: | Hi!
I'm a gentoo noob so plz go easy on me.
|
Let's face it, we are all n00b's...
| tiny wrote: | I'm in control of 3 machines all running Linux. One of them could be
my Gentoo gateway, since bandwidth over the day is very poor and
downloading would require a lot of time.
But this machine isn't running Gentoo distribution and it needs to stay this
way (for now). The only thing Im looking for is fast network access to
ebuild packages. Portage can be on the machine I'm installing Gentoo to.
|
OK what you want are the files from the "distfiles" directory. I agree to that, since they can be quite large, up to several hundreds of MB. BTW, the total size is 21 GB right now!
| tiny wrote: | Question is how can I keep packages on my server machine (gateway)
up to date?
Lets say I put them in apache directories. What is the simplest method of
keeping those packages up to date on regular bases?
I was thinking in this way ... simple bash script with wget would do it.
I decide which packages I wont, I DL them and later on I keep
them updated. Only how would I keep them updated so I dont burn a lot of
bandwith. Is there posibilty to use rsync from my Linux distribution?
Or something like that.
Storage space isn't problem.
Oh BTW, how much space would take if I DL all the packages. I'm not
gonna, I'm just wondering.
regards,
Tiny |
I started trying to do this from Windows 2000 but soon gave up on the idea. "rsync" would seem to be the most reliable method, and I don't see why it wouldn't work from other Linux distro's, but I don't know for sure. Problem is to limit the distfiles directory size, however.
The nicest solution seems to be the proxy method as described elesewhere in this thread, but I have no experience, I just mirror the whole lot. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Dec 03, 2003 7:52 am Post subject: |
|
|
| tiny wrote: |
But this machine isn't running Gentoo distribution and it needs to stay this
way (for now). The only thing Im looking for is fast network access to
ebuild packages. Portage can be on the machine I'm installing Gentoo to.
Question is how can I keep packages on my server machine (gateway)
up to date?
Lets say I put them in apache directories. What is the simplest method of
keeping those packages up to date on regular bases?
|
This sounds like tsp-cache will also work for you. It only requires apache on the cache machine, it doesn't have to run gentoo. The other gentoo machines will automatically keep the cache up to date. tsp-cache will ensure you only download packages once. Once in the cache, packages will be delivered at lan speeds. Check out
https://forums.gentoo.org/viewtopic.php?t=110973 |
|
| Back to top |
|
|
roofy
Tux's lil' helper
Joined: 01 Sep 2003
Posts: 86
Location: miami
|
| Posted: Wed Dec 03, 2003 5:06 pm Post subject: |
|
|
DAMN! you beat me to it.
i had the same exact system set up at work, where we have a cluster of 3 portage servers that compile -b pakages and feed then to 300 clients, and it automatically syncs and then scripts control the automatic emerge world/system, i even have notify by email working.....i was going to write a doc on the whole thing...kudos
_________________
/* event horizon */
Athlon XP 2100+ (o/c'ed to 2.1ghz)
LanParty NForce2 Ultra
80GB Maxtor HD
48x CD-RW
ATI 9700 Pro |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
