Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Sep 07, 2004 12:13 pm    Post subject: Reply with quote

First, let me say that if you own all other boxes nfs sharing is an option. There are corruption and other issues you need to worry about but if you can control all the boxes then you can minimize the chance of corruption.

I don't want to turn this thread into a flamewar about the potential corruption and other problems with sharing the distfile dir. I'd ask that someone start another thread to discuss that issue if anyone has any questions.

This thread is about the advantages of Http-Replicator!!

Http-Replicator is an easy and quick install that relieves the users of virtually all headaches associated with multiple boxen. Once installed, there are no corruption or other issues to worry about. Any box can emerge any package at any time and http-replicator will make sure the downloads happen in a safe and efficient manner!!

Think about this advantage in a situation like a college campus: If you have 10 (or 100) avid gentooers that start a mozilla upgrade at nearly the same time what will happen with nfs sharing?

All 10 or (100) boxes will download a separate copy of the 34438800 byte file!! Why? Because each box will look for the package and will see an incomplete download and start a new download to resume the file from the already heavily loaded volunteer mirrors.

If Http-Replicator were is use, it would intercept all 10 (or 100) download requests from the LAN and would simultaneously download the file from the internet and stream the file to all 10 (or 100) users. The result would be only 1 copy of the package downloaded from the internet and all LAN clients simultaneously receiving the package!! Any client that requests the package after the download is complete will receive the file from the cache at LAN speeds!

The next question you have to ask yourself is what were you thinking when your tried to nfs share your distfile dir with 10 (or 100) semi-strangers on your college campus!!
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Tue Sep 07, 2004 7:59 pm    Post subject: Reply with quote

I agree totally. Http-replicator is the right tool for the job. Sharing over NFS is a horrible hack that will bite you eventually (but I did use it for a long time until I discovered HR).
Back to top
View user's profile Send private message
MattSharp
Apprentice
Apprentice


Joined: 15 Aug 2003
Posts: 258

PostPosted: Tue Sep 14, 2004 4:03 am    Post subject: Reply with quote

Im sure someone has had this problem but I didn't see anything about it. When I try to emerge packages on the server, it has problems downloading the package:

Quote:

lancelot root # emerge mutt
Calculating dependencies ...done!
>>> emerge (1 of 1) mail-client/mutt-1.5.6-r3 to /
>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo//distfiles/mutt-1.5.6i.tar.gz
--23:02:56-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving gentoo.chem.wisc.edu... 128.104.70.13
Connecting to gentoo.chem.wisc.edu[128.104.70.13]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,910,730 [application/x-tar]

100%[=================>] 2,910,730 153.54K/s ETA 00:00

23:03:14 (156.60 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
--23:03:15-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving gentoo.chem.wisc.edu... 128.104.70.13
Connecting to gentoo.chem.wisc.edu[128.104.70.13]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,910,730 [application/x-tar]

100%[=================>] 2,910,730 155.56K/s ETA 00:00

23:03:33 (156.05 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
--23:03:33-- http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving mirrors.tds.net... 216.170.153.145
Connecting to mirrors.tds.net[216.170.153.145]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,910,730 [application/x-gzip]

100%[=================>] 2,910,730 360.87K/s ETA 00:00

23:03:47 (223.50 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
--23:03:47-- ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving ftp.mutt.org... 82.165.34.161
Connecting to ftp.mutt.org[82.165.34.161]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /mutt/devel ... done.
==> PASV ... done. ==> RETR mutt-1.5.6i.tar.gz ... done.
Length: 2,910,730 (unauthoritative)

100%[=========================================>] 2,910,730 208.38K/s ETA 00:00

23:04:01 (232.11 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730]

!!! Couldn't download mutt-1.5.6i.tar.gz. Aborting.


What is causing this?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Sep 14, 2004 4:52 am    Post subject: Reply with quote

MattSharp wrote:
Im sure someone has had this problem but I didn't see anything about it. When I try to emerge packages on the server, it has problems downloading the package:



Your connections are all on port 80, so your not downloading the package through http-replicator.... Are you trying to or is this a general gentoo problem your having?
Back to top
View user's profile Send private message
MattSharp
Apprentice
Apprentice


Joined: 15 Aug 2003
Posts: 258

PostPosted: Tue Sep 14, 2004 3:37 pm    Post subject: Reply with quote

flybynite wrote:
MattSharp wrote:
Im sure someone has had this problem but I didn't see anything about it. When I try to emerge packages on the server, it has problems downloading the package:



Your connections are all on port 80, so your not downloading the package through http-replicator.... Are you trying to or is this a general gentoo problem your having?


I am trying to use http-replicator. This is the "server" box. The clients seem to work fine...I think.

But when I try to get a package on there I have that problem. I commented out the http_proxy line cause that didn't work either. Here is what I get when I try it with that line:

Quote:

Calculating dependencies ...done!
>>> emerge (1 of 1) mail-client/mutt-1.5.6-r3 to /
>>> Previously fetched file: patch-1.5.6.rr.compressed.gz MD5 ;-)
>>> Previously fetched file: patch-1.5.6.dw.pgp-timeout.1 MD5 ;-)
>>> Previously fetched file: patch-1.5.6.tg.hcache.0 MD5 ;-)
>>> Previously fetched file: patch-1.5.5.1.cd.edit_threads.9.5-gentoo.bz2 MD5 ;-
)
>>> Previously fetched file: patch-1.5.6.dw.mbox-hook.1 MD5 ;-)
>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo//distfiles/mutt-1.5.6i.tar.gz
--10:27:28-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,910,730

100%[====================================>] 2,910,730 2.23M/s

10:27:30 (2.22 MB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
--10:27:30-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,910,730

100%[====================================>] 2,910,730 --.--K/s

10:27:30 (38.12 MB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
--10:27:30-- http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,910,730

100%[====================================>] 2,910,730 --.--K/s

10:27:30 (32.61 MB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
--10:27:30-- ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving ftp.mutt.org... 82.165.34.161
Connecting to ftp.mutt.org[82.165.34.161]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /mutt/devel ... done.
==> PASV ... done. ==> RETR mutt-1.5.6i.tar.gz ... done.
Length: 2,910,730 (unauthoritative)

100%[====================================>] 2,910,730 361.11K/s ETA 00:00

10:27:40 (339.14 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730]

!!! Couldn't download mutt-1.5.6i.tar.gz. Aborting.

!!! Fetch for /usr/portage/mail-client/mutt/mutt-1.5.6-r3.ebuild failed, continu
ing...



!!! Some fetch errors were encountered. Please see above for details.


So what am I doing wrong? One thing I did notice. If I try to emerge mutt on one of the "clients" it downloads it from the "server" and works fine. But I can't emerge it on the server. What is the problem?

Also, one scenario I also have is that one of my clients is a desktop and it's not always on the network. Is there an easy way to make it not try to use the server when its no on the network? Maybe comment something out or something? Or maybe write into it, that if the server is missing try something else?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Sep 15, 2004 1:19 pm    Post subject: Reply with quote

MattSharp wrote:
I am trying to use http-replicator.
...
I commented out the http_proxy line
So what am I doing wrong?


If your using the standard cache and distfile locations, try this:
Code:

mv /var/cache/http-replicator/*  /usr/portage/distfiles/
/usr/bin/repcacheman
rm /usr/portage/distfiles/*


MattSharp wrote:

Also, one scenario I also have is that one of my clients is a desktop and it's not always on the network. Is there an easy way to make it not try to use the server when its no on the network?


What I do is use Quickswitch for my laptop (emerge quickswitch). Quickswitch is designed to allow multiple configs for different networks with different services. I have a setting that switches my /etc/make.conf when I move off my lan.

More info here: http://www.newsforge.com/article.pl?sid=01/12/22/2118213&mode=thread

Homepage http://muthanna.com/quickswitch

How to change on boot with gentoo https://forums.gentoo.org/viewtopic.php?t=96281&highlight=quickswitch+boot
Back to top
View user's profile Send private message
Master One
l33t
l33t


Joined: 25 Aug 2003
Posts: 754
Location: Austria

PostPosted: Wed Sep 15, 2004 4:51 pm    Post subject: Reply with quote

That's so great! Also I was fighting with some strange problem, now it seems to be working just fine.

Nevertheless someone please tell me:
Quote:
Don't forget that portage needs mirrors! Edit GENTOO_MIRRORS in /etc/make.conf to add more http mirrors and place any ftp mirrors LAST. The default mirrors in gentoo leave something to be desired Smile Use mirrorselect if you need help in selecting mirrors.

Is GENTOO_MIRRORS to be set on all machines (server+workstations), or only on the server???

Quote:
Also, some packages in portage have a RESTRICT="nomirror" option which will prevent portage from checking replicator for those packages. The following will override this behavior. Create the file "/etc/portage/mirrors" containing: local http://gentoo.oregonstate.edu

Is this mirrors file to be created on all machines (server+workstations), or only on the server???

I find this mirrors thing a little confusing, because why should the workstations bother about that, if only the server is intended to download and cache the distfiles.

My last question: Why is nothing showing up in the http-replicator log-file??? I can see the file gets generated on startup, but until now I could never find any info in there, also I did some successfull downloads (indeed the file has always a size of zero).
_________________
Las torturas mentales de la CIA
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Wed Sep 15, 2004 8:57 pm    Post subject: Reply with quote

The mirrors need to be specified for every user of the proxy, since a request is sent to the proxy for a file on a particular mirror, and the proxy 'does it's thing' - it's meant to be fairly transparent to the client. I use the same set of mirrors for every node in the network, including the server.

The clients need the mirrors because the server allows the client to download the file 'by proxy'.

As for the log file - check the permissions are OK.
Back to top
View user's profile Send private message
Master One
l33t
l33t


Joined: 25 Aug 2003
Posts: 754
Location: Austria

PostPosted: Fri Sep 17, 2004 1:10 pm    Post subject: Reply with quote

meowsqueak wrote:
The mirrors need to be specified for every user of the proxy, since a request is sent to the proxy for a file on a particular mirror, and the proxy 'does it's thing' - it's meant to be fairly transparent to the client. I use the same set of mirrors for every node in the network, including the server. The clients need the mirrors because the server allows the client to download the file 'by proxy'. As for the log file - check the permissions are OK.

Ok, I've set GENTOO_MIRRORS and /etc/portage/mirrors now on all machines (server + workstations).

It's definitely working fine, but I still could not find out, why the logfile keeps beeing empty. I tried it with chown root:root and portage:portage, but this does not change anything (in http-replicator.conf there is explicitely mentioned, that no write access has to be granted anyway). The logfile definitely gets created on startup, if I delete it before, but until now, nothing showed up in there... :?
_________________
Las torturas mentales de la CIA
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Sep 17, 2004 2:40 pm    Post subject: Reply with quote

Master One wrote:

It's definitely working fine, but I still could not find out, why the logfile keeps beeing empty.


The stable version of http-replicator writes the log file in a very lazy way. It won't actually write the log entries to disk till you fill the buffer. It uses less system resources this way on a busy server but it can take a while to flush to disk on a home lan :-) This has been changed in the latest version of http-replicator which flushes every log entry to disk.

So if your just trying to make sure your setup is OK, it probably is. If you want to see the logs now, just use the telnet monitor and you can see the messages as they happen! Check your config to make sure it is enabled and then just telnet to the port to see any messages in real time!
Back to top
View user's profile Send private message
Master One
l33t
l33t


Joined: 25 Aug 2003
Posts: 754
Location: Austria

PostPosted: Fri Sep 17, 2004 4:44 pm    Post subject: Reply with quote

Thanx, flybynite, as everthing is working, I just wanted to check, if the log confirms that. The less resources it takes, the better. Hopefully it will make it into portage soon now, so that I can benefit from the ongoing development.
_________________
Las torturas mentales de la CIA
Back to top
View user's profile Send private message
jdoe
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jun 2003
Posts: 122
Location: Italy - Florence - Empoli

PostPosted: Sat Sep 18, 2004 3:27 pm    Post subject: Reply with quote

hi..
i'm trying to set up http-rep, for my 3 client lan.. I followed the how-to, but i have some problem on client side.
On the server everything works fine, i can see downloading at 15MB/s a cached file.
On client side i have this error:

Code:
Calculating dependencies ...done!
>>> emerge (1 of 1) net-fs/samba-3.0.7-r1 to /
>>> Resuming download...
>>> Downloading http://samba.idealx.org/dist/smbldap-tools-0.8.5.tgz
--17:39:32--  http://samba.idealx.org/dist/smbldap-tools-0.8.5.tgz
           => `/usr/portage/distfiles/distfiles/smbldap-tools-0.8.5.tgz'
Resolving slave... 192.168.1.3
Connecting to slave[192.168.1.3]:12000... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
17:39:52 ERROR 503: Service Unavailable.

!!! Couldn't download smbldap-tools-0.8.5.tgz. Aborting.


here is conf files:
client make.conf:
Code:
GENTOO_MIRRORS="http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/ http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/"
http_proxy="http://slave:12000"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"


server make.conf
Code:

http_proxy="http://127.0.0.1:12000"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
GENTOO_MIRRORS="http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/ http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/"


/etc/http-replicator.conf
Code:

PORT = 12000
IP = ['127.0.0.1','192.168.1.2','192.168.1.3']


where i wrong?

Thanks, jdoe
_________________
me
Back to top
View user's profile Send private message
lisa
Retired Dev
Retired Dev


Joined: 01 Jun 2003
Posts: 273
Location: York, UK again! Horray!

PostPosted: Sat Sep 18, 2004 7:42 pm    Post subject: Reply with quote

For distfile sharing I've set up Samba. It's way easier. For Rsync I have my server set up as a rsync mirror for all of my internal Gentoo machines.
_________________
Distcc guide
Visit my website
I maintain Distcc, Ccache, Memcached, and some others (i think)
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Sat Sep 18, 2004 8:52 pm    Post subject: Reply with quote

jdoe wrote:
server make.conf
Code:

http_proxy="http://127.0.0.1:12000"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
GENTOO_MIRRORS="http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/ http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/"


/etc/http-replicator.conf
Code:

PORT = 12000
IP = ['127.0.0.1','192.168.1.2','192.168.1.3']


where i wrong?

Thanks, jdoe


change your http_proxy="http://127.0.0.1:12000" to either your external ip or to 192.168.1.3 (which would be your lan ip), i.e. to http_proxy="http://192.168.1.3:12000
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Sep 20, 2004 9:31 pm    Post subject: Reply with quote

lisa wrote:
For distfile sharing I've set up Samba.


That doesn't solve the concurrency problem either, does it?
Back to top
View user's profile Send private message
jdoe
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jun 2003
Posts: 122
Location: Italy - Florence - Empoli

PostPosted: Tue Sep 21, 2004 6:38 am    Post subject: Reply with quote

BlinkEye wrote:

change your http_proxy="http://127.0.0.1:12000" to either your external ip or to 192.168.1.3 (which would be your lan ip), i.e. to http_proxy="http://192.168.1.3:12000


thanks, it works :)
It' really a nice tool...
_________________
me
Back to top
View user's profile Send private message
fritz
n00b
n00b


Joined: 15 Aug 2004
Posts: 7

PostPosted: Sun Oct 03, 2004 4:16 pm    Post subject: cpu usage Reply with quote

it's working but cpu usage is ~25% continuously on server while downloading with single client
doesnt' matter if the file is already in the cache or not

think this is rather high for just downloading, am i the only one with this problem or is this lack of effiency in http-replicator?

cpu is:
model name : AMD Duron(tm) Processor
cpu MHz : 1016.568
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Oct 05, 2004 10:27 pm    Post subject: Re: cpu usage Reply with quote

fritz wrote:

think this is rather high for just downloading, am i the only one with this problem or is this lack of effiency in http-replicator?


I have an even slower box and I don't see cpu that high while maxing out a 100Mbs lan. cpu usage is dependent on many factors though. http-replicator is usually waiting on your disk or network. If I had to guess I'd say that you need to check your network config - especially your network card module or driver. I've heard of some drivers that have two versions that work but with high cpu usage on one. Could also be as simple as you don't have dma on your disks.

Start checking by transferring some data on your lan with ftp or http and watching the cpu usage.
Back to top
View user's profile Send private message
fritz
n00b
n00b


Joined: 15 Aug 2004
Posts: 7

PostPosted: Thu Oct 07, 2004 6:33 pm    Post subject: Reply with quote

indeed transfering a large file through nfs also takes 10-15% cpu usage, which also seems rather high. dma is on though, and i'm using the sis900 driver, any issues with that one?

hmm it's an onboard-nic, maybe that got something to do with it?

cheers
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Oct 07, 2004 7:38 pm    Post subject: Reply with quote

fritz wrote:
i'm using the sis900 driver, any issues with that one?


A quick google shows a few possibilities that may or may not apply to your exact motherboard/kernel combination.

Most point towards 2.6.X problems with that chipset while 2.4.X works better. The problem seems to be sharing pci irq's which shows up in high demand usage of both the disk and the network driver. Try copying a large file to another disk or to /dev/null and see it the cpu is still high.

Some fixes I've seen recommend making sure to disable the Plug and Play support in the motherboard BIOS.

Check you dmesg for errors also.

Sorry I couldn't give you an easy fix.....
Back to top
View user's profile Send private message
fritz
n00b
n00b


Joined: 15 Aug 2004
Posts: 7

PostPosted: Thu Oct 07, 2004 9:48 pm    Post subject: Reply with quote

using 2.6.8-gentoo, no errors in dmesg and the sis900 gets it own irq. what's weird is cat'ing a file to /dev/null takes 15% usage, using dd 30% 8O (prob not related to my problem, just weird). is this too high because i get similar results on another computer?

appreciate the help and the fast reply :)
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Oct 08, 2004 5:43 am    Post subject: Reply with quote

Those results would seem to say your level of cpu usage is normal to your box and http-replicator is very efficient, using the same cpu as cat'ing to /dev/null.....
Back to top
View user's profile Send private message
fritz
n00b
n00b


Joined: 15 Aug 2004
Posts: 7

PostPosted: Fri Oct 08, 2004 4:03 pm    Post subject: Reply with quote

flybynite wrote:
Those results would seem to say your level of cpu usage is normal to your box and http-replicator is very efficient, using the same cpu as cat'ing to /dev/null.....


ooooooh :oops: indeed, see your point

guess i'll just accept it takes 25% of my cpu then, i'll try to play a bit with hdparm
Back to top
View user's profile Send private message
piyo
n00b
n00b


Joined: 20 Mar 2004
Posts: 12

PostPosted: Sat Oct 09, 2004 12:37 pm    Post subject: Reply with quote

The instructions say nothing about what to clean periodically in the cache directory.

Here I delete anything that hasn't been referenced in the last two weeks.

Code:
colinux root # cat /etc/cron.daily/http-replicator
#!/bin/sh
/bin/nice /usr/bin/find /var/cache/http-replicator -type f -ctime +14 | \
    /usr/bin/xargs --no-run-if-empty /usr/bin/rm -f
/bin/nice /usr/bin/repcacheman

---
piyo
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 09, 2004 7:31 pm    Post subject: Reply with quote

piyo wrote:
The instructions say nothing about what to clean periodically in the cache directory.


Thanks for the script piyo!

I've talked about this before in the thread. The truth is it is much harder than it looks. That's the reason gentoo doesn't have a default way of cleaning distfiles either, many people need different things.

Most scripts in this board that clean distfiles will work with http-replicators cache. You can find scripts to clean the cache based on any number of criteria.

Your script is nicely done. I like the way you nice'd the tasks and you remembered to still run repcacheman.

The one problem I see with your script is that many gentooers have followed the Gentoo Install Handbook's recommendation of using `noatime` in fstab for performance reasons.
Quote:

Now, to improve performance, most users would want to add the noatime option as mountoption, which results in a faster system since access times aren't registered (you don't need those generally anyway):

From http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=8#doc_chap1


With noatime as a mount option, your script would simply delete files older than two weeks regardless of how recently they were served to clients.......
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 22, 23, 24  Next
Page 6 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum