View previous topic :: View next topic |
Author |
Message |
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Mon Dec 01, 2003 5:46 am Post subject: HOWTO: tsp-cache - Streaming Distfile Cache for LAN Use |
|
|
If you have multiple Gentoo boxen on a lan your probably looking for some speed/bandwidth saving options. All the methods seem to have some advantages and disadvantages.
The options include:
1. complete distfile mirror - 99% waste of bandwidth for yourself and the Gentoo mirrors
2. squid proxy cache - expires distfiles - really only meant for short term storage
3. network share of the /usr/portage/distfiles tree - Portage was not designed to share this directory live. Risk of one client downloading the package while another tries to use that half downloaded file. What happens when multiple clients all download the same file at the same time?
4. partial rsync gateway ( HOWTO: Central Gentoo Mirror for your Internal Network )
How do you get the files each client needs to the gateway? Running an emerge -f on the gateway doesn't help download all files because of different use settings the clients may have. If multiple clients request the same file not in the cache the client emerge will fail and keep failing until the whole file is downloaded on the gateway.
There may be other options I haven't found yet or workaround for some of these problems, I propose another solution.
tsp-cache - a Turbo Streaming Proxy cache for distfiles
It is a proxy designed for distfiles. Files are served from the cache first. If not in the cache the proxy will simultaneously download the file from a mirror and stream the file to all clients requesting the file while the download is in progress.
Advantages:
1. Bandwidth savings. The file is only downloaded once no matter how many clients request the file.
2. No Portage problems. tsp-cache is transparent to clients. tsp-cache is just listed as one of the many download mirrors in make.conf. Failure of the tsp-cache host machine will not cripple the clients. They will automatically use the next available mirror preserving current behavior.
3. Streaming to clients. tsp-cache will download the file from a mirror while simultaneously streaming the file to multiple clients. Therefore, client emerges will not fail even though the files are not in the cache.
4. Reports. tsp-cache can create html reports of cache efficiency and bandwidth.
Disadvantages?
1. Apache required. If you running more than two machines you probably have an apache server available. You may get other servers to work.
2. Separate cache. tsp-cache saves files with headers attached for speedy streaming. These files are not directly usable by the host system. It may be better to keep the cache separate rather than giving the user "apache" read/write access to your distfile directory. Future versions could use the distfile directory if this proves valuable.
Installation:
download http://www.updatedlinux.com/tsp-cache/tsp-cache.tar.bz2
extract the files
Code: |
tar -xjvf tsp-cache.bz2
|
Code: | Listing of tsp-cache files:
tsp-cache-apache.conf - append to /etc/apache/conf/apache.conf to activate tsp-cache
tsp-cache.conf - general settings
tsp-cache.pl - the script
tsp-cache-lib.pl - parses configfile
tsp-cache-report.pl - perl script to generate reports - should be run by cron daily
report_cron - generates reports when run, copy to /etc/cron.daily/
|
Edit the apache.conf,tsp-cache.conf, report_cron files if you want to use other directories.
Create these directories mode 755 owned by apache:apache and create blank access.log and error.log files.
/var/cache/tsp-cache/
/var/cache/tsp-cache/log/
/var/cache/tsp-cache/private/
Code: |
mkdir -p -m 755 /var/cache/tsp-cache/private
mkdir -p -m 755 /var/cache/tsp-cache/log
touch /var/cache/tsp-cache/log/error.log
touch /var/cache/tsp-cache/log/access.log
chown -Rc apache:apache /var/cache/tsp-cache
|
ls -l /var/cache/
drwxr-xr-x 4 apache apache 4096 Nov 30 20:31 tsp-cache
ls -l /var/cache/tsp-cache/
drwxr-xr-x 2 apache apache 4096 Nov 30 17:42 log
drwxr-xr-x 2 apache apache 4096 Nov 30 20:31 private
ls -l /var/cache/tsp-cache/log/
-rw-r--r-- 1 apache apache 2012 Nov 30 20:31 access.log
-rw-r--r-- 1 apache apache 299 Nov 30 19:52 error.log
Create this directory mode 755 owned by root
/usr/share/tsp-cache
Code: |
mkdir -m 755 /usr/share/tsp-cache
|
copy tsp-cache-lib.pl tsp-cache-report.pl tsp-cache.conf tsp-cache.pl to /usr/share/tsp-cache
ls -l /usr/share/tsp-cache/
-rwxr-xr-x 1 root root 1143 Nov 30 14:38 tsp-cache-lib.pl
-rwxr-xr-x 1 root root 5396 Nov 30 15:11 tsp-cache-report.pl
-rw-r--r-- 1 root root 2126 Nov 30 15:05 tsp-cache.conf
-rwxr--r-x 1 root root 20498 Nov 30 15:04 tsp-cache.pl
Add the contents of tsp-cache-apache.conf to the real apache.conf
Code: |
cat tsp-cache-apache.conf >> /etc/apache/conf/apache.conf
|
restart apache
Code: |
/etc/init.d/apache restart
|
Create reports daily if you wish
copy report_cron to /etc/cron.daily/
Point your browser to http://your.cache.com/tsp-cache to see verify it's working and see some config info.
Try http://your.cache.com/tsp-cache/report to see the daily stat report
Client Setup
edit the client's make.conf to uncomment the GENTOO_MIRRORS line and prefix some mirrors with your box/tsp-cache/ - assuming the tsp-cache server is www.yourcache.com
Using other mirrors is OK.. A problem with large files from www.ibiblio.org has been reported. I recommend you use another mirror till this gets corrected.
Original setting:
Code: |
#GENTOO_MIRRORS="<your_mirror_here> http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo"
|
New Setting:
Code: |
GENTOO_MIRRORS="http://www.yourcache.com/tsp-cache/gentoo.oregonstate.edu http://www.yourcache.com/tsp-cache/www.gtlib.cc.gatech.edu/pub/gentoo"
|
Enjoy
Last edited by flybynite on Tue Dec 16, 2003 11:42 pm; edited 1 time in total |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Wed Dec 03, 2003 12:33 pm Post subject: |
|
|
Excellent initiative!
This is exactly what I've been looking for since I started to use Gentoo more seriously.
May I add two comments?
- Don't you think the file tsp-cache.conf belongs somewhere under /etc?
- When do we see an ebuild of tsp-cache?
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Wed Dec 03, 2003 4:08 pm Post subject: Ebuild and Config |
|
|
Yes, I think the config should be under /etc!! I just wanted to get some feedback first. Does anyone care if I create a /etc/tsp-cache/ directory or should I just move the file to /etc/tsp-cache.conf? I think the directory is the best option since the future might hold more features and another config file.
Ebuilds should be easy since I worked hard to make the howto very specific on the install. I do need some feedback from the gentoo devs. I'm not sure how best to handle the apache.conf changes to make an emerge unmerge go cleanly. I think the best way is to follow the lead of the mod_php module and surround everything with <IfDefine TSP-CACHE>
</IfDefine> but I'll have to do some reasearch to see if that will work. |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Wed Dec 03, 2003 4:12 pm Post subject: Re: Ebuild and Config |
|
|
flybynite wrote: | I think the directory is the best option since the future might hold more features and another config file.
|
I fully agree. It's a clean approach. But then again, I'm not part of the Gentoo development team.
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
masked n00b
Joined: 15 Aug 2003 Posts: 2
|
Posted: Thu Dec 04, 2003 11:07 pm Post subject: etc etc |
|
|
I am confused I followed the instructions i can view http://host/tsp-cache from the clients, i have fed the data but i still get HTML error 500 when contacting the server the access.log and error.log contain nothing.
any idea whats going wrong? _________________ there should be something for me to say here ... :S |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Dec 05, 2003 5:19 am Post subject: Set Dubug=1 |
|
|
Looke in tsp-cache.conf and change the debug level from 0 to 1. It's the last config item.
debug=0
to
debug=1
that should help shed some light on whats happening.
Did you change any of the directories? Make sure that /var/cache/tsp-cache are writable by apache or the user your web server is running as.
ls -l /var/cache/tsp-cache/
drwxr-xr-x 2 apache apache 4096 Nov 30 17:42 log
drwxr-xr-x 2 apache apache 4096 Nov 30 20:31 private
The files access.log and error.log must exist in the log directory. The script will not create them if they don't already exist. |
|
Back to top |
|
|
nixnick n00b
Joined: 29 Nov 2002 Posts: 9
|
Posted: Wed Dec 10, 2003 8:19 pm Post subject: excellent work! |
|
|
This in combination with the local rsync mirror detailed at here makes for a very effective bandwidth saver. It's kind to both the gentoo mirrors and to my local connection. I just set it up and tried it out and it works great. thanks.
One issue, though. You can point the tsp-cache 'server' to use the local cache as well via 'GENTOO_MIRRORS' . however you will have duplicate files in /usr/portage/distfiles and /var/cache/tsp-cache. symlinking and/or changing DISTDIR to /var/cache/tsp-cache results in md5 checksum mismatches. The short term solution is to use a nightly cron job to clear out /usr/portage/distfiles/ indiscriminately on the cache server. But I'm wondering if there's a more elegant way to elminate that duplication of distfiles....
Additionally you may want to keep your /var/cach/tsp-cache clean of out-of-date distfiles. the script found here can be easily modified to meet the need. Just change the line that says:
Code: |
distdir = portage.config().environ()['DISTDIR']
|
to
Code: |
distdir = '/var/cache/tsp-cache'
|
and save it somewhere in root's path as cacheclean.py . Add a small script to /etc/cron.daily/dist-cache-clean ie
Code: |
#!/bin/sh
rm -r /usr/portage/distfiles/*
/usr/local/bin/cacheclean.py
|
Should keep you from wasting diskspace unnecessarily on the cache server |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Thu Dec 11, 2003 2:58 am Post subject: Perfect Combo!!!!!!! |
|
|
You should point all gentoo boxen, including the tsp-cache host, to tsp-cache. That gives the most benefits.
Yes, tsp-cache along with a rsync server is the perfect combo!! It lets me download updated packages to multiple boxes at LAN speeds, thats why I put it together.
tsp-cache does require a separate cache. You may have missed this in the original post. I did this for a couple of reasons. Separate cache _may_ temporarily cost a bit of disk space but is probably the best choice for now.
I do plan to get an ebuild together and look at different options for the cache.
I would like some feedback to help select the best option. The options so far to eliminate the duplicate files on the host are:
1. move packages from the cache to the hosts /distfiles directory after the download is complete. The problem with this is tsp-cache doesn't have enough file permissions to do this since it is running as the user apache. Default /distfile permissions will deny writing to the user apache.
Giving the apache user read/write permission in the hosts /distfile directory has security implications. This may not be a good idea at all if used on a campus wide lan or more public system. Probably not a problem for a private lan if you trust all the clients.
2. Simply delete the packages in the hosts /distfile directory when done with the emerge. No wasted space, no security implications, packages still instantly available though tsp-cache when needed by the host or any other machine. Seems like the most elegant solution to me right now. Really these files are the wasted space if you think about it.
It wouldn't be hard to create a script that runs from cron that moves files from the cache to /distfiles. I'm just not convinced this is better than just deleting the hosts /distfiles for most users.
Think of any more options/comments? |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Thu Dec 11, 2003 8:52 am Post subject: Re: Perfect Combo!!!!!!! |
|
|
flybynite wrote: | Simply delete the packages in the hosts /distfile directory when done with the emerge. |
I feel that cleaning out the distfiles directory is out of scope for tps-cache. IMHO, tsp-cache should know nothing about the distfiles directory.
I also think that emerge should have an option in make.conf that you could turn on to always clean out the distfiles after a compilation/installation. That way anyone maintaining a local caching mirror could decide for himself if s/he wants to have the distfiles cleaned and redo the download when/if necessary. But this idea has so far had few followers.
Let tsp-cache deal with tsp-cache's cache and leave the distfiles problem to emerge.
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Thu Dec 11, 2003 9:50 am Post subject: |
|
|
I've had a few situations when tsp-cache has delivered files that have had MD5 checksum errors and/or wrong total file size.
This has happened for quite large files, like 100MB+, and tsp-cache is currently run in a RAM challenged PC. It is possible that adding more RAM might help. (I'm currently hanging out on E-Bay to get more RAM to this old buddy.)
Has anyone else noticed the same type of symptoms using tsp-cache?
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Thu Dec 11, 2003 7:19 pm Post subject: Probably a slow mirror.... |
|
|
Try changing your mirror.
tsp-cache uses wget to download the file - very robust and proven. All tsp-cache does is pass along the data to clients so there are fewer problems.
I think its normal to see some incomplete downloads when the gentoo mirror your using is overloaded. This was a big problem when everyone was using one mirror. Now that we have more mirrors to choose from its not as bad.
emerge mirrorselect to help you find more mirrors. If you don't change your mirror, your stuck downloading from the same mirror everyone else is using.
When wget encounters a problem with a mirror tsp-cache does its best to pass along the error. Check your error.log as see if it caught any problems with wget downloading the file in question.
Make sure your not trying to use the tsp-cache'd file from the tsp-cache directory. The files tsp-cache saves to disk are not usable except by tsp-cache. They contain http headers to enable streaming to multiple clients while tsp-cache itself is still downloading the file. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Thu Dec 11, 2003 7:37 pm Post subject: Apache |
|
|
If your low on memory, you might look into minimizing apache memory usage. It starts a whole bunch of servers to handle heavy loads. If your only using it locally for just a few boxes you might try some of these options. I'm not saying these settings are best for you, some apache gurus could have some better options, but if your low on memory, you gotta make some cuts somewhere...
# listen on my lan only, I don't serve the public
BindAddress 192.168.2.184
# I know the load on my network, I don't have to prepare for spikes
# so why keep extras hanging around?
MinSpareServers 1
MaxSpareServers 2
# Number of servers to start initially --- should be a reasonable ballpark
# apache will make more if needed
StartServers 2
# Limit on total number of servers running, i.e., limit on the number
# of clients who can simultaneously connect --- if this limit is ever
# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
# It is intended mainly as a brake to keep a runaway server from taking
# the system with it as it spirals down...
#
MaxClients 10
I only have 5 machines on my lan and they don't sync at once. |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Thu Dec 11, 2003 8:17 pm Post subject: Re: Apache |
|
|
flybynite wrote: | If your low on memory, you might look into minimizing apache memory usage. |
This will make a lot of sense even when I have more memory in this PC. Just from the type of usage of this server, it makes sense to put in these restrictions.
Thanks a lot
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
gonna n00b
Joined: 07 Dec 2003 Posts: 11 Location: Sydney
|
Posted: Fri Dec 12, 2003 9:47 am Post subject: |
|
|
Biker wrote: | I've had a few situations when tsp-cache has delivered files that have had MD5 checksum errors and/or wrong total file size.
This has happened for quite large files, like 100MB+, and tsp-cache is currently run in a RAM challenged PC. It is possible that adding more RAM might help. (I'm currently hanging out on E-Bay to get more RAM to this old buddy.)
Has anyone else noticed the same type of symptoms using tsp-cache?
Biker |
I have been getting the same issues with MD5 checksums. I have noticed if it only happens when the tsp-cache does not have the file requested in the cache. When you remove the file from "/usr/tmp/portage/" and run emerge again it is fine. The retrieved files on the caching server are fine, it only occurs on the client machine. I also have noticed this happens with larger files.
Greg |
|
Back to top |
|
|
gonna n00b
Joined: 07 Dec 2003 Posts: 11 Location: Sydney
|
Posted: Fri Dec 12, 2003 9:57 am Post subject: Re: Probably a slow mirror.... |
|
|
flybynite wrote: | Try changing your mirror.
tsp-cache uses wget to download the file - very robust and proven. All tsp-cache does is pass along the data to clients so there are fewer problems.
I think its normal to see some incomplete downloads when the gentoo mirror your using is overloaded. This was a big problem when everyone was using one mirror. Now that we have more mirrors to choose from its not as bad.
|
The checksum problem is only occuring on the client requesting the files where they are not already cached and is only occuring sometimes. Files on the caching server are always correct.
I have it running under a Debian box which is also running apt-cacher, so I doubt it is related to apache.
Hope this feedback helps. It is a very nice program and I have already saved plently of bandwidth with it.
Greg |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Fri Dec 12, 2003 10:43 am Post subject: |
|
|
flybynite:
I could repeatedly create the MD5 problem by running: emerge -f openoffice
I'm using the european mirror, so should get a new server each time through the round-robin mechanism.
The 180+ MB file never came down correctly. (The bandwith gain was quickly lost since I run this command maybe 10 times.
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Dec 12, 2003 5:00 pm Post subject: |
|
|
gonna wrote: |
When you remove the file from "/usr/tmp/portage/" and run emerge again it is fine. Greg |
Are you sure mean /usr/tmp/portage/? I believe portage checks md5 before copying to this working directory. Not a problem if you just wrote the wrong directory. I'll try to get to the bottom of this.
Please show me the output of emerge -f openoffice or the file your having problems with.
Please show me the /var/cache/tsp-cache/log/error.log. You can trim to just the days in question if it is large.
Same for you Biker. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Dec 12, 2003 5:16 pm Post subject: |
|
|
Biker wrote: | flybynite:
I'm using the european mirror, so should get a new server each time through the round-robin mechanism.
Biker |
I know of no _download_ mirror rotation. Rsync mirrors use a rotation, download mirrrors don't.
show me the output of emerge -f openoffice and post your /var/cache/tsp-cache/log/error.log - trim it to the days in question if large for some reason. |
|
Back to top |
|
|
gonna n00b
Joined: 07 Dec 2003 Posts: 11 Location: Sydney
|
Posted: Sat Dec 13, 2003 1:10 pm Post subject: |
|
|
flybynite wrote: | gonna wrote: |
When you remove the file from "/usr/tmp/portage/" and run emerge again it is fine. Greg |
Are you sure mean /usr/tmp/portage/? I believe portage checks md5 before copying to this working directory. Not a problem if you just wrote the wrong directory. I'll try to get to the bottom of this. |
I got the directory wrong, I meant the '/usr/portage/distfiles' directory. |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Sun Dec 14, 2003 8:37 pm Post subject: |
|
|
flybynite wrote: |
I know of no _download_ mirror rotation. Rsync mirrors use a rotation, download mirrrors don't. |
Sorry. Me bad. You're right.
flybynite wrote: |
show me the output of emerge -f openoffice and post your /var/cache/tsp-cache/log/error.log - trim it to the days in question if large for some reason. |
emerge -f app-office/openoffice
Code: |
>>> Downloading http://tjalve/yggdrasil.home/tsp-cache/gentoo.oregonstate.edu/distfiles/OOo_1.1.0_source.tar.bz2
--19:21:55-- http://tjalve/yggdrasil.home/tsp-cache/gentoo.oregonstate.edu/distfiles/OOo_1.1.0_source.tar.bz2
=> `/usr/portage/distfiles/OOo_1.1.0_source.tar.bz2'
Resolving tjalve... 10.0.0.8
Connecting to tjalve[10.0.0.8]:80... connected.
HTTP request sent, awaiting response... 404 Not Found
19:21:55 ERROR 404: Not Found.
>>> Downloading http://tjalve.yggdrasil.home/tsp-cache/www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/OOo_1.1.0_source.tar.bz2
--19:21:55-- http://tjalve.yggdrasil.home/tsp-cache/www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/OOo_1.1.0_source.tar.bz2
=> `/usr/portage/distfiles/OOo_1.1.0_source.tar.bz2'
Resolving tjalve.yggdrasil.home... 10.0.0.8
Connecting to tjalve.yggdrasil.home[10.0.0.8]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 189,164,050 [application/x-tar]
100%[==============================================================================================>] 189,164,050 37.95K/s ETA 00:00
21:13:07 (27.70 KB/s) - `/usr/portage/distfiles/OOo_1.1.0_source.tar.bz2' saved [189164050/189164050]
!!! Fetched file: OOo_1.1.0_source.tar.bz2 MD5 FAILED! Removing corrupt distfile...
>>> Downloading http://gentoo.oregonstate.edu/distfiles/OOo_1.1.0_source.tar.bz2
|
From which point the file is successfully downloaded directly from gentoo.oregonstate.edu.
Edit: The first error, the 404, comes from me having incorrectly configured the client to use tjalve/yggdrasil.home as a server instead of tjalve.yggdrasil.home. Correcting this didn't change anything, though.
Now lets walk over to the tsp-cache server:
Code: |
# ls -l /var/cache/tsp-cache/log/error.log
-rw-r--r-- 1 root root 0 Dec 14 19:19 /var/cache/tsp-cache/log/error.log
|
Note the zero size of the error.log file.
The last line in /var/log/apache2/error_log (not sure if this is related to the problem in question):
Code: |
[Sun Dec 14 21:13:07 2003] [error] [client 10.0.0.13] Use of uninitialized value in concatenation (.) or string at /usr/share/tsp-cache/tsp-cache.pl
line 180.
|
No more entries in this log around this time.
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Dec 16, 2003 11:06 pm Post subject: FOUND!!! |
|
|
Thanks for the report Biker.
Before your post of the emerge -f, I tried to duplicate the problem you reported, but couldn't.
After looking at your report, I could. Here is the fix
I tried to download openoffice source from ibiblio more than 20 times and NEVER SUCCEDED!!!
I tried different times of day, using both using tsp-cache and wget alone. Wget always failed. It doesn't seem to report any errors that tsp-cache can trap. Sometimes wget failed at 20k and sometimes at 178 MB, but it always failed. Its also interesting that some other mirrors don't even have this file.
I don't know whether ibiblio has a corrupt copy of this file or whether it just can't seem to deliver it to wget. Shorter files do seem to work fine from ibiblio. Maybe ibiblio has some sort of timeout since it only fails on this huge, long download.
Tsp-cache doesn't md5 the files. Since wget reported no errors the corrupt file is still part of the cache tsp-cache is using. Delete the openoffice source file from /var/cache/tsp-cache/ to purge this corrupt file.
I'll look into this more, this gives me some ideas on making tsp-cache more error tolerant, but the bottom line is, if wget can't get the file from ibiblio, ibiblio is broke..... |
|
Back to top |
|
|
Biker Apprentice
Joined: 11 Jun 2003 Posts: 170 Location: A very dark, cold and moisty place...
|
Posted: Wed Dec 17, 2003 7:55 am Post subject: Re: FOUND!!! |
|
|
flybynite wrote: | Thanks for the report Biker. |
Anytime. Providing "complaints" is easy. You're the one fixing things. So: Thank you!
flybynite wrote: |
I tried to download openoffice source from ibiblio more than 20 times and NEVER SUCCEDED!!!
|
And I thought that I spent a lot of time trying this out...
flybynite wrote: | I'll look into this more, this gives me some ideas on making tsp-cache more error tolerant |
A noble goal. I'll be happy to try braking your code. (I'm quite good at that.
flybynite wrote: | if wget can't get the file from ibiblio, ibiblio is broke..... |
Makes sense to me.
Biker _________________ The Internet never forgets.
Where 'never' points in the direction of a moment in the very, very far future. |
|
Back to top |
|
|
Nekkrist n00b
Joined: 09 Oct 2003 Posts: 33
|
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Dec 26, 2003 11:43 pm Post subject: sorry, |
|
|
Sorry, tsp-cache doesn't work with ftp mirrors. I bet it could be done but I haven't looked into it.
Most mirrors I've seen support both http and ftp. They support http because http is easier to secure, and easily cache-able.
What chanages to the install were needed for apache2? Please share.... I wanted to add apache2 to the install instructions but don't want to install apache2 just to know the changes. |
|
Back to top |
|
|
Nekkrist n00b
Joined: 09 Oct 2003 Posts: 33
|
Posted: Sat Dec 27, 2003 4:05 am Post subject: |
|
|
Well at first I thought I would have to install apache or do some hacking to get it to work with apache2 because I got an error 500 trying to check if it was up and running. All i really had to do is read the directions again and realize that I had forgotten to copy a file over
So basically it just works, I didn't change anything and it hasn't failed yet whiile running the apache2 server. |
|
Back to top |
|
|
|