Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: tsp-cache - Streaming Distfile Cache for LAN Use
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Dec 01, 2003 5:46 am    Post subject: HOWTO: tsp-cache - Streaming Distfile Cache for LAN Use Reply with quote

If you have multiple Gentoo boxen on a lan your probably looking for some speed/bandwidth saving options. All the methods seem to have some advantages and disadvantages.

The options include:

1. complete distfile mirror - 99% waste of bandwidth for yourself and the Gentoo mirrors

2. squid proxy cache - expires distfiles - really only meant for short term storage

3. network share of the /usr/portage/distfiles tree - Portage was not designed to share this directory live. Risk of one client downloading the package while another tries to use that half downloaded file. What happens when multiple clients all download the same file at the same time?

4. partial rsync gateway ( HOWTO: Central Gentoo Mirror for your Internal Network )

How do you get the files each client needs to the gateway? Running an emerge -f on the gateway doesn't help download all files because of different use settings the clients may have. If multiple clients request the same file not in the cache the client emerge will fail and keep failing until the whole file is downloaded on the gateway.


There may be other options I haven't found yet or workaround for some of these problems, I propose another solution.

tsp-cache - a Turbo Streaming Proxy cache for distfiles

It is a proxy designed for distfiles. Files are served from the cache first. If not in the cache the proxy will simultaneously download the file from a mirror and stream the file to all clients requesting the file while the download is in progress.


Advantages:
1. Bandwidth savings. The file is only downloaded once no matter how many clients request the file.

2. No Portage problems. tsp-cache is transparent to clients. tsp-cache is just listed as one of the many download mirrors in make.conf. Failure of the tsp-cache host machine will not cripple the clients. They will automatically use the next available mirror preserving current behavior.

3. Streaming to clients. tsp-cache will download the file from a mirror while simultaneously streaming the file to multiple clients. Therefore, client emerges will not fail even though the files are not in the cache.

4. Reports. tsp-cache can create html reports of cache efficiency and bandwidth.

Disadvantages?

1. Apache required. If you running more than two machines you probably have an apache server available. You may get other servers to work.

2. Separate cache. tsp-cache saves files with headers attached for speedy streaming. These files are not directly usable by the host system. It may be better to keep the cache separate rather than giving the user "apache" read/write access to your distfile directory. Future versions could use the distfile directory if this proves valuable.


Installation:

download http://www.updatedlinux.com/tsp-cache/tsp-cache.tar.bz2

extract the files
Code:

tar -xjvf tsp-cache.bz2


Code:
 Listing of tsp-cache files:

tsp-cache-apache.conf - append to /etc/apache/conf/apache.conf to activate tsp-cache
tsp-cache.conf - general settings
tsp-cache.pl - the script
tsp-cache-lib.pl - parses configfile
tsp-cache-report.pl - perl script to generate reports - should be run by cron daily
report_cron - generates reports when run, copy to /etc/cron.daily/


Edit the apache.conf,tsp-cache.conf, report_cron files if you want to use other directories.

Create these directories mode 755 owned by apache:apache and create blank access.log and error.log files.

/var/cache/tsp-cache/
/var/cache/tsp-cache/log/
/var/cache/tsp-cache/private/

Code:

mkdir -p -m 755 /var/cache/tsp-cache/private
mkdir -p -m 755 /var/cache/tsp-cache/log
touch /var/cache/tsp-cache/log/error.log
touch /var/cache/tsp-cache/log/access.log
chown -Rc apache:apache /var/cache/tsp-cache


ls -l /var/cache/
drwxr-xr-x 4 apache apache 4096 Nov 30 20:31 tsp-cache
ls -l /var/cache/tsp-cache/
drwxr-xr-x 2 apache apache 4096 Nov 30 17:42 log
drwxr-xr-x 2 apache apache 4096 Nov 30 20:31 private
ls -l /var/cache/tsp-cache/log/
-rw-r--r-- 1 apache apache 2012 Nov 30 20:31 access.log
-rw-r--r-- 1 apache apache 299 Nov 30 19:52 error.log


Create this directory mode 755 owned by root
/usr/share/tsp-cache

Code:

mkdir -m 755 /usr/share/tsp-cache


copy tsp-cache-lib.pl tsp-cache-report.pl tsp-cache.conf tsp-cache.pl to /usr/share/tsp-cache
ls -l /usr/share/tsp-cache/
-rwxr-xr-x 1 root root 1143 Nov 30 14:38 tsp-cache-lib.pl
-rwxr-xr-x 1 root root 5396 Nov 30 15:11 tsp-cache-report.pl
-rw-r--r-- 1 root root 2126 Nov 30 15:05 tsp-cache.conf
-rwxr--r-x 1 root root 20498 Nov 30 15:04 tsp-cache.pl

Add the contents of tsp-cache-apache.conf to the real apache.conf
Code:

cat tsp-cache-apache.conf >> /etc/apache/conf/apache.conf


restart apache
Code:

/etc/init.d/apache restart


Create reports daily if you wish

copy report_cron to /etc/cron.daily/

Point your browser to http://your.cache.com/tsp-cache to see verify it's working and see some config info.
Try http://your.cache.com/tsp-cache/report to see the daily stat report

Client Setup

edit the client's make.conf to uncomment the GENTOO_MIRRORS line and prefix some mirrors with your box/tsp-cache/ - assuming the tsp-cache server is www.yourcache.com

Using other mirrors is OK.. A problem with large files from www.ibiblio.org has been reported. I recommend you use another mirror till this gets corrected.

Original setting:
Code:

#GENTOO_MIRRORS="<your_mirror_here> http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo"

New Setting:
Code:

GENTOO_MIRRORS="http://www.yourcache.com/tsp-cache/gentoo.oregonstate.edu http://www.yourcache.com/tsp-cache/www.gtlib.cc.gatech.edu/pub/gentoo"

Enjoy


Last edited by flybynite on Tue Dec 16, 2003 11:42 pm; edited 1 time in total
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Wed Dec 03, 2003 12:33 pm    Post subject: Reply with quote

Excellent initiative!

This is exactly what I've been looking for since I started to use Gentoo more seriously.


May I add two comments?

    - Don't you think the file tsp-cache.conf belongs somewhere under /etc?

    - When do we see an ebuild of tsp-cache? ;-)


Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Dec 03, 2003 4:08 pm    Post subject: Ebuild and Config Reply with quote

Yes, I think the config should be under /etc!! I just wanted to get some feedback first. Does anyone care if I create a /etc/tsp-cache/ directory or should I just move the file to /etc/tsp-cache.conf? I think the directory is the best option since the future might hold more features and another config file.

Ebuilds should be easy since I worked hard to make the howto very specific on the install. I do need some feedback from the gentoo devs. I'm not sure how best to handle the apache.conf changes to make an emerge unmerge go cleanly. I think the best way is to follow the lead of the mod_php module and surround everything with <IfDefine TSP-CACHE>
</IfDefine> but I'll have to do some reasearch to see if that will work.
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Wed Dec 03, 2003 4:12 pm    Post subject: Re: Ebuild and Config Reply with quote

flybynite wrote:
I think the directory is the best option since the future might hold more features and another config file.


I fully agree. It's a clean approach. But then again, I'm not part of the Gentoo development team. ;-)

Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
masked
n00b
n00b


Joined: 15 Aug 2003
Posts: 2

PostPosted: Thu Dec 04, 2003 11:07 pm    Post subject: etc etc Reply with quote

:) I am confused I followed the instructions i can view http://host/tsp-cache from the clients, i have fed the data but i still get HTML error 500 when contacting the server the access.log and error.log contain nothing.

any idea whats going wrong?
_________________
there should be something for me to say here ... :S
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Dec 05, 2003 5:19 am    Post subject: Set Dubug=1 Reply with quote

Looke in tsp-cache.conf and change the debug level from 0 to 1. It's the last config item.


debug=0
to
debug=1

that should help shed some light on whats happening.

Did you change any of the directories? Make sure that /var/cache/tsp-cache are writable by apache or the user your web server is running as.

ls -l /var/cache/tsp-cache/
drwxr-xr-x 2 apache apache 4096 Nov 30 17:42 log
drwxr-xr-x 2 apache apache 4096 Nov 30 20:31 private

The files access.log and error.log must exist in the log directory. The script will not create them if they don't already exist.
Back to top
View user's profile Send private message
nixnick
n00b
n00b


Joined: 29 Nov 2002
Posts: 9

PostPosted: Wed Dec 10, 2003 8:19 pm    Post subject: excellent work! Reply with quote

This in combination with the local rsync mirror detailed at here makes for a very effective bandwidth saver. It's kind to both the gentoo mirrors and to my local connection. I just set it up and tried it out and it works great. thanks.

One issue, though. You can point the tsp-cache 'server' to use the local cache as well via 'GENTOO_MIRRORS' . however you will have duplicate files in /usr/portage/distfiles and /var/cache/tsp-cache. symlinking and/or changing DISTDIR to /var/cache/tsp-cache results in md5 checksum mismatches. The short term solution is to use a nightly cron job to clear out /usr/portage/distfiles/ indiscriminately on the cache server. But I'm wondering if there's a more elegant way to elminate that duplication of distfiles....

Additionally you may want to keep your /var/cach/tsp-cache clean of out-of-date distfiles. the script found here can be easily modified to meet the need. Just change the line that says:
Code:

distdir = portage.config().environ()['DISTDIR']

to
Code:

distdir = '/var/cache/tsp-cache'

and save it somewhere in root's path as cacheclean.py . Add a small script to /etc/cron.daily/dist-cache-clean ie
Code:

#!/bin/sh

rm -r /usr/portage/distfiles/*
/usr/local/bin/cacheclean.py


Should keep you from wasting diskspace unnecessarily on the cache server
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Dec 11, 2003 2:58 am    Post subject: Perfect Combo!!!!!!! Reply with quote

You should point all gentoo boxen, including the tsp-cache host, to tsp-cache. That gives the most benefits.

Yes, tsp-cache along with a rsync server is the perfect combo!! It lets me download updated packages to multiple boxes at LAN speeds, thats why I put it together.

tsp-cache does require a separate cache. You may have missed this in the original post. I did this for a couple of reasons. Separate cache _may_ temporarily cost a bit of disk space but is probably the best choice for now.

I do plan to get an ebuild together and look at different options for the cache.

I would like some feedback to help select the best option. The options so far to eliminate the duplicate files on the host are:

1. move packages from the cache to the hosts /distfiles directory after the download is complete. The problem with this is tsp-cache doesn't have enough file permissions to do this since it is running as the user apache. Default /distfile permissions will deny writing to the user apache.

Giving the apache user read/write permission in the hosts /distfile directory has security implications. This may not be a good idea at all if used on a campus wide lan or more public system. Probably not a problem for a private lan if you trust all the clients.

2. Simply delete the packages in the hosts /distfile directory when done with the emerge. No wasted space, no security implications, packages still instantly available though tsp-cache when needed by the host or any other machine. Seems like the most elegant solution to me right now. Really these files are the wasted space if you think about it.

It wouldn't be hard to create a script that runs from cron that moves files from the cache to /distfiles. I'm just not convinced this is better than just deleting the hosts /distfiles for most users.

Think of any more options/comments?
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Thu Dec 11, 2003 8:52 am    Post subject: Re: Perfect Combo!!!!!!! Reply with quote

flybynite wrote:
Simply delete the packages in the hosts /distfile directory when done with the emerge.


I feel that cleaning out the distfiles directory is out of scope for tps-cache. IMHO, tsp-cache should know nothing about the distfiles directory.

I also think that emerge should have an option in make.conf that you could turn on to always clean out the distfiles after a compilation/installation. That way anyone maintaining a local caching mirror could decide for himself if s/he wants to have the distfiles cleaned and redo the download when/if necessary. But this idea has so far had few followers.

Let tsp-cache deal with tsp-cache's cache ;-) and leave the distfiles problem to emerge.

Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Thu Dec 11, 2003 9:50 am    Post subject: Reply with quote

I've had a few situations when tsp-cache has delivered files that have had MD5 checksum errors and/or wrong total file size.

This has happened for quite large files, like 100MB+, and tsp-cache is currently run in a RAM challenged PC. It is possible that adding more RAM might help. (I'm currently hanging out on E-Bay to get more RAM to this old buddy.)

Has anyone else noticed the same type of symptoms using tsp-cache?


Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Dec 11, 2003 7:19 pm    Post subject: Probably a slow mirror.... Reply with quote

Try changing your mirror.

tsp-cache uses wget to download the file - very robust and proven. All tsp-cache does is pass along the data to clients so there are fewer problems.

I think its normal to see some incomplete downloads when the gentoo mirror your using is overloaded. This was a big problem when everyone was using one mirror. Now that we have more mirrors to choose from its not as bad.

emerge mirrorselect to help you find more mirrors. If you don't change your mirror, your stuck downloading from the same mirror everyone else is using.

When wget encounters a problem with a mirror tsp-cache does its best to pass along the error. Check your error.log as see if it caught any problems with wget downloading the file in question.

Make sure your not trying to use the tsp-cache'd file from the tsp-cache directory. The files tsp-cache saves to disk are not usable except by tsp-cache. They contain http headers to enable streaming to multiple clients while tsp-cache itself is still downloading the file.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Dec 11, 2003 7:37 pm    Post subject: Apache Reply with quote

If your low on memory, you might look into minimizing apache memory usage. It starts a whole bunch of servers to handle heavy loads. If your only using it locally for just a few boxes you might try some of these options. I'm not saying these settings are best for you, some apache gurus could have some better options, but if your low on memory, you gotta make some cuts somewhere...

# listen on my lan only, I don't serve the public
BindAddress 192.168.2.184

# I know the load on my network, I don't have to prepare for spikes
# so why keep extras hanging around?
MinSpareServers 1
MaxSpareServers 2

# Number of servers to start initially --- should be a reasonable ballpark
# apache will make more if needed
StartServers 2

# Limit on total number of servers running, i.e., limit on the number
# of clients who can simultaneously connect --- if this limit is ever
# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
# It is intended mainly as a brake to keep a runaway server from taking
# the system with it as it spirals down...
#
MaxClients 10

I only have 5 machines on my lan and they don't sync at once.
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Thu Dec 11, 2003 8:17 pm    Post subject: Re: Apache Reply with quote

flybynite wrote:
If your low on memory, you might look into minimizing apache memory usage.


This will make a lot of sense even when I have more memory in this PC. Just from the type of usage of this server, it makes sense to put in these restrictions.

Thanks a lot
Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
gonna
n00b
n00b


Joined: 07 Dec 2003
Posts: 11
Location: Sydney

PostPosted: Fri Dec 12, 2003 9:47 am    Post subject: Reply with quote

Biker wrote:
I've had a few situations when tsp-cache has delivered files that have had MD5 checksum errors and/or wrong total file size.

This has happened for quite large files, like 100MB+, and tsp-cache is currently run in a RAM challenged PC. It is possible that adding more RAM might help. (I'm currently hanging out on E-Bay to get more RAM to this old buddy.)

Has anyone else noticed the same type of symptoms using tsp-cache?
Biker


I have been getting the same issues with MD5 checksums. I have noticed if it only happens when the tsp-cache does not have the file requested in the cache. When you remove the file from "/usr/tmp/portage/" and run emerge again it is fine. The retrieved files on the caching server are fine, it only occurs on the client machine. I also have noticed this happens with larger files.

Greg
Back to top
View user's profile Send private message
gonna
n00b
n00b


Joined: 07 Dec 2003
Posts: 11
Location: Sydney

PostPosted: Fri Dec 12, 2003 9:57 am    Post subject: Re: Probably a slow mirror.... Reply with quote

flybynite wrote:
Try changing your mirror.

tsp-cache uses wget to download the file - very robust and proven. All tsp-cache does is pass along the data to clients so there are fewer problems.

I think its normal to see some incomplete downloads when the gentoo mirror your using is overloaded. This was a big problem when everyone was using one mirror. Now that we have more mirrors to choose from its not as bad.


The checksum problem is only occuring on the client requesting the files where they are not already cached and is only occuring sometimes. Files on the caching server are always correct.

I have it running under a Debian box which is also running apt-cacher, so I doubt it is related to apache.

Hope this feedback helps. It is a very nice program and I have already saved plently of bandwidth with it.

Greg
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Fri Dec 12, 2003 10:43 am    Post subject: Reply with quote

flybynite:

I could repeatedly create the MD5 problem by running: emerge -f openoffice

I'm using the european mirror, so should get a new server each time through the round-robin mechanism.

The 180+ MB file never came down correctly. (The bandwith gain was quickly lost since I run this command maybe 10 times. ;-)

Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Dec 12, 2003 5:00 pm    Post subject: Reply with quote

gonna wrote:

When you remove the file from "/usr/tmp/portage/" and run emerge again it is fine. Greg


Are you sure mean /usr/tmp/portage/? I believe portage checks md5 before copying to this working directory. Not a problem if you just wrote the wrong directory. I'll try to get to the bottom of this.

Please show me the output of emerge -f openoffice or the file your having problems with.

Please show me the /var/cache/tsp-cache/log/error.log. You can trim to just the days in question if it is large.


Same for you Biker.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Dec 12, 2003 5:16 pm    Post subject: Reply with quote

Biker wrote:
flybynite:
I'm using the european mirror, so should get a new server each time through the round-robin mechanism.
Biker


I know of no _download_ mirror rotation. Rsync mirrors use a rotation, download mirrrors don't.

show me the output of emerge -f openoffice and post your /var/cache/tsp-cache/log/error.log - trim it to the days in question if large for some reason.
Back to top
View user's profile Send private message
gonna
n00b
n00b


Joined: 07 Dec 2003
Posts: 11
Location: Sydney

PostPosted: Sat Dec 13, 2003 1:10 pm    Post subject: Reply with quote

flybynite wrote:
gonna wrote:

When you remove the file from "/usr/tmp/portage/" and run emerge again it is fine. Greg


Are you sure mean /usr/tmp/portage/? I believe portage checks md5 before copying to this working directory. Not a problem if you just wrote the wrong directory. I'll try to get to the bottom of this.


I got the directory wrong, I meant the '/usr/portage/distfiles' directory.
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Sun Dec 14, 2003 8:37 pm    Post subject: Reply with quote

flybynite wrote:

I know of no _download_ mirror rotation. Rsync mirrors use a rotation, download mirrrors don't.

Sorry. Me bad. You're right. :oops:

flybynite wrote:

show me the output of emerge -f openoffice and post your /var/cache/tsp-cache/log/error.log - trim it to the days in question if large for some reason.


emerge -f app-office/openoffice
Code:

>>> Downloading http://tjalve/yggdrasil.home/tsp-cache/gentoo.oregonstate.edu/distfiles/OOo_1.1.0_source.tar.bz2
--19:21:55--  http://tjalve/yggdrasil.home/tsp-cache/gentoo.oregonstate.edu/distfiles/OOo_1.1.0_source.tar.bz2
           => `/usr/portage/distfiles/OOo_1.1.0_source.tar.bz2'
Resolving tjalve... 10.0.0.8
Connecting to tjalve[10.0.0.8]:80... connected.
HTTP request sent, awaiting response... 404 Not Found
19:21:55 ERROR 404: Not Found.
 
>>> Downloading http://tjalve.yggdrasil.home/tsp-cache/www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/OOo_1.1.0_source.tar.bz2



--19:21:55--  http://tjalve.yggdrasil.home/tsp-cache/www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/OOo_1.1.0_source.tar.bz2
           => `/usr/portage/distfiles/OOo_1.1.0_source.tar.bz2'
Resolving tjalve.yggdrasil.home... 10.0.0.8
Connecting to tjalve.yggdrasil.home[10.0.0.8]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 189,164,050 [application/x-tar]
 
100%[==============================================================================================>] 189,164,050   37.95K/s    ETA 00:00
 
21:13:07 (27.70 KB/s) - `/usr/portage/distfiles/OOo_1.1.0_source.tar.bz2' saved [189164050/189164050]
 
!!! Fetched file: OOo_1.1.0_source.tar.bz2 MD5 FAILED! Removing corrupt distfile...
>>> Downloading http://gentoo.oregonstate.edu/distfiles/OOo_1.1.0_source.tar.bz2

From which point the file is successfully downloaded directly from gentoo.oregonstate.edu.
Edit: The first error, the 404, comes from me having incorrectly configured the client to use tjalve/yggdrasil.home as a server instead of tjalve.yggdrasil.home. Correcting this didn't change anything, though. :-(

Now lets walk over to the tsp-cache server:
Code:

# ls -l /var/cache/tsp-cache/log/error.log
-rw-r--r--    1 root     root            0 Dec 14 19:19 /var/cache/tsp-cache/log/error.log

Note the zero size of the error.log file.


The last line in /var/log/apache2/error_log (not sure if this is related to the problem in question):
Code:

[Sun Dec 14 21:13:07 2003] [error] [client 10.0.0.13] Use of uninitialized value in concatenation (.) or string at /usr/share/tsp-cache/tsp-cache.pl
line 180.

No more entries in this log around this time.

Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Dec 16, 2003 11:06 pm    Post subject: FOUND!!! Reply with quote

Thanks for the report Biker.

Before your post of the emerge -f, I tried to duplicate the problem you reported, but couldn't.

After looking at your report, I could. Here is the fix

Code:
REMOVE www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/ from the GENTOO_MIRROR on all your clients!  Then remove the corrupt file from /var/cache/tsp-cache/


I tried to download openoffice source from ibiblio more than 20 times and NEVER SUCCEDED!!!

I tried different times of day, using both using tsp-cache and wget alone. Wget always failed. It doesn't seem to report any errors that tsp-cache can trap. Sometimes wget failed at 20k and sometimes at 178 MB, but it always failed. Its also interesting that some other mirrors don't even have this file.

I don't know whether ibiblio has a corrupt copy of this file or whether it just can't seem to deliver it to wget. Shorter files do seem to work fine from ibiblio. Maybe ibiblio has some sort of timeout since it only fails on this huge, long download.

Tsp-cache doesn't md5 the files. Since wget reported no errors the corrupt file is still part of the cache tsp-cache is using. Delete the openoffice source file from /var/cache/tsp-cache/ to purge this corrupt file.

I'll look into this more, this gives me some ideas on making tsp-cache more error tolerant, but the bottom line is, if wget can't get the file from ibiblio, ibiblio is broke.....
Back to top
View user's profile Send private message
Biker
Apprentice
Apprentice


Joined: 11 Jun 2003
Posts: 170
Location: A very dark, cold and moisty place...

PostPosted: Wed Dec 17, 2003 7:55 am    Post subject: Re: FOUND!!! Reply with quote

flybynite wrote:
Thanks for the report Biker.

Anytime. Providing "complaints" is easy. You're the one fixing things. So: Thank you!


flybynite wrote:

I tried to download openoffice source from ibiblio more than 20 times and NEVER SUCCEDED!!!

And I thought that I spent a lot of time trying this out... ;-)

flybynite wrote:
I'll look into this more, this gives me some ideas on making tsp-cache more error tolerant

A noble goal. I'll be happy to try braking your code. (I'm quite good at that. ;-)

flybynite wrote:
if wget can't get the file from ibiblio, ibiblio is broke.....

Makes sense to me.
Biker
_________________
The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future.
Back to top
View user's profile Send private message
Nekkrist
n00b
n00b


Joined: 09 Oct 2003
Posts: 33

PostPosted: Tue Dec 23, 2003 9:40 pm    Post subject: Reply with quote

Woah! Great work!

Now, is there any way to get this to work with the ftp servers such as ftp://mirror.iawnet.sandia.gov/pub/gentoo/

All I get is (as I was expecting) a 404 error when trying to access http://my.tspcache.com/mirror.iawnet.sandia.gov/pub/gentoo/ via portage

Otherwise, it all works great! Apache2 fuctions with your scripts correctly from what I can see. :)

Nekkrist
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Dec 26, 2003 11:43 pm    Post subject: sorry, Reply with quote

Nekkrist wrote:

Now, is there any way to get this to work with the ftp servers such as ftp://mirror.iawnet.sandia.gov/pub/gentoo/
Nekkrist


Sorry, tsp-cache doesn't work with ftp mirrors. I bet it could be done but I haven't looked into it.

Most mirrors I've seen support both http and ftp. They support http because http is easier to secure, and easily cache-able.

What chanages to the install were needed for apache2? Please share.... I wanted to add apache2 to the install instructions but don't want to install apache2 just to know the changes.
Back to top
View user's profile Send private message
Nekkrist
n00b
n00b


Joined: 09 Oct 2003
Posts: 33

PostPosted: Sat Dec 27, 2003 4:05 am    Post subject: Reply with quote

Well at first I thought I would have to install apache or do some hacking to get it to work with apache2 because I got an error 500 trying to check if it was up and running. All i really had to do is read the directions again and realize that I had forgotten to copy a file over :oops:

So basically it just works, I didn't change anything and it hasn't failed yet whiile running the apache2 server.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum