| View previous topic :: View next topic |
| Author |
Message |
woZa
Guru
Joined: 18 Nov 2003
Posts: 340
Location: The Steel City - UK
|
 Posted: Wed Aug 04, 2004 9:37 pm Post subject: |
 |
|
thanks flybynite...
_________________
A few months struggling with gentoo is better than a lifetime struggling with windoze! |
|
| Back to top |
|
 |
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Thu Aug 05, 2004 5:47 am Post subject: You asked for External proxy support!! |
|
|
New version ready for testing!! Although the init code,logging, and options are cleaned up, this version primarily adds external proxy support.
Current users not needing external proxy support should wait for the next stable release. Experienced users can help test.
If your don't have any version of http-replicator installed and wish to help test, follow the HOWTO first, then upgrade to this latest version.
1. Download and install the ebuild in the portage overlay directory (/usr/local/portage by default), then emerge http-replicator.
Download:
http-replicator-flybynite-1.5.tar.bz2
which contains both the old and new ebuild's.
| Code: |
# cd /usr/local/portage
# tar -xvjf /PathToFile/http-replicator-flybynite-1.5.tar.bz2
|
Then emerge:
| Code: |
# emerge -u http-replicator
|
This will upgrade http-replicator.
Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults.
**Note repcacheman has changed. If you changed the default cache dir or the default user, you must call repcacheman with those options.
| Code: |
repcacheman --user USER --dir /path/to/cache
|
2. You can downgrade http-replicator if you have any problems with the new version
| Code: |
# emerge =net-misc/http-replicator-2.0-r2
|
Please report your experience!!
[Edit] update to rc3
Last edited by flybynite on Fri Aug 06, 2004 7:32 pm; edited 2 times in total |
|
| Back to top |
|
|
Boworr
n00b
Joined: 30 Jul 2004
Posts: 45
Location: England. Where the history comes from.
|
| Posted: Thu Aug 05, 2004 3:06 pm Post subject: Manually adding items to the cache? |
|
|
Hi,
Firstly, this is great and saves me a huge amount of time/bandwidth at home.
Is it possible to manually add items to the cache? For example, when I'm at work I'd like to use the companies bandwidth  to fetch all my packages, then when I get home copy them over the cache server so the other machines can get them quickly.
Is this possible? If so, what do I copy (where does emerge store temp downloads anyway?) and where do I copy it to?
Thanks |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Thu Aug 05, 2004 3:40 pm Post subject: |
|
|
Yes, you can manually add files to the cache. The included script repcacheman will do it for your also.
The safest method would be to copy the files to /usr/portage/distfiles and then run repcacheman. repcacheman is a script designed to import files that were downloaded from ftp or manually added. The script will verify the md5sum of all files before adding them to replicators cache.
The fastest method would be to just copy the files directly to replicators cache which is /var/cache/http-replicator unless you changed the defaults. The only drawback to this method is that the md5sum's aren't checked. If any files your downloaded had errors in them you will have to manually delete them from the cache.
The latest version of replicator will md5sum replicators cache, it's just still in testing. |
|
| Back to top |
|
|
meowsqueak
Veteran
Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand
|
| Posted: Thu Aug 05, 2004 7:52 pm Post subject: Re: You asked for External proxy support!! |
|
|
| flybynite wrote: | | New version ready for testing!! |
Excellent! I will definitely test this out and let you know how it goes. Thanks! |
|
| Back to top |
|
|
Boworr
n00b
Joined: 30 Jul 2004
Posts: 45
Location: England. Where the history comes from.
|
| Posted: Thu Aug 05, 2004 9:38 pm Post subject: |
|
|
| flybynite wrote: | | Yes, you can manually add files to the cache. The included script repcacheman will do it for your also. |
So where do I copy them from?  i.e, where does emerge store files with I --fetch them?
Boworr |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Thu Aug 05, 2004 10:39 pm Post subject: |
|
|
Emerge stores all the files in /usr/portage/distfiles
Copy them from there to the /usr/portage/distfiles on the box with replicator and then run repcacheman |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Fri Aug 06, 2004 12:14 am Post subject: |
|
|
Hello, ok trying to save some disk space on my note book and decrease bandwidth for emerge sync etc i setup rsync mirror and http-replicator... very nice but i have some question as i now plan to build a small local server to be the main rsync/http-replicator.
1. getting rid of /usr/portage on clients, currently i have nfs exported the /usr/portage from server to client... is this good idea? would love to get rid of /usr/portage dir on local machines and just have /usr/portage/http-replocator cache on server
2. disk space and dir partitions for server - with the server i plan to have a seperate partition for portage files, from what i can gather /usr/portage is not that important or large, with http-replicator and the reprocache on cron moving the files to http-replicator cache dir, and it is this cache dir that i can give its own partition.
Am i correct?
cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Fri Aug 06, 2004 12:46 am Post subject: |
|
|
umm, seems that nfs exporting /usr/portage to client does not work 
rsync mirror fails and emege -uD world just seems to go on for ever finding dependancies.
h
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
meowsqueak
Veteran
Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand
|
| Posted: Fri Aug 06, 2004 4:11 am Post subject: |
|
|
| /usr/portage should be stored somewhere FAST and accessing it over NFS is relatively slow. Every time Portage builds a dependency tree, it has to read lots of files from here. |
|
| Back to top |
|
|
JohnHerdy
n00b
Joined: 11 Jul 2002
Posts: 61
|
| Posted: Fri Aug 06, 2004 8:11 am Post subject: Re: You asked for External proxy support!! |
|
|
| flybynite wrote: | New version ready for testing!! Although the init code,logging, and options are cleaned up, this version primarily adds external proxy support.
|
yahoo!!!, I will test it and report back, thanks a lot!!! |
|
| Back to top |
|
|
JohnHerdy
n00b
Joined: 11 Jul 2002
Posts: 61
|
| Posted: Fri Aug 06, 2004 12:15 pm Post subject: |
|
|
| I'm testing the proxy support. Can you tell me where to add my username and password? |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Aug 06, 2004 7:48 pm Post subject: |
|
|
New http-replicator-2.1_rc3 Includes support for basic proxy authentication (as requested by JohnHerdy)
Everybody testing please update to the latest version and test. You will need to download http-replicator-flybynite-1.5.tar.bz2 Instructions are in this post:
https://forums.gentoo.org/viewtopic.php?t=173226&start=76 |
|
| Back to top |
|
|
JohnHerdy
n00b
Joined: 11 Jul 2002
Posts: 61
|
| Posted: Fri Aug 06, 2004 8:49 pm Post subject: |
|
|
| flybynite wrote: | | New http-replicator-2.1_rc3 Includes support for basic proxy authentication (as requested by JohnHerdy) |
Man you have no idea how happy you have made me. It's working like a charm!!!
Minor improvement;
- if the cachedir doesn't exists when the daemon starts create it and set the correct permissions or else the daemon crashes.
- in the default /etc/conf.d/http-replicator add an " at the end of the --external examples, because without the " at the end http-replicator crashes. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Fri Aug 06, 2004 9:25 pm Post subject: |
|
|
Your welcome!!
| JohnHerdy wrote: |
Minor improvement;
- if the cachedir doesn't exist when the daemon starts create it and set the correct permissions or else the daemon crashes.
|
Actually, replicator will print 'invalid directory' or 'no read/write permission for directory' and then exit
You might have missed it because the instructions are only in the complete HOWTO at the start of this thread, but...
I wrote the repcacheman utility to install http-replicator. It will install and maintain replicators cache. It creates the cache dir if it doesn't exist and then transfers files from the host to the cache.
If your server box is gentoo, you should still run repcacheman to prime the cache with your existing files.
In this release candidate version, if you changed the cache dir from the default /var/cache/http-replicator, or changed the user from portage, you need to set the cache dir and/or the user on the command line like so
| Code: |
# /usr/bin/repcacheman --dir /var/cache/http-replicator --user portage
|
| JohnHerdy wrote: |
- in the default /etc/conf.d/http-replicator add an " at the end of the --external examples, because without the " at the end http-replicator crashes. |
Thanks, my typo... Fixed... |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Sun Aug 08, 2004 8:25 am Post subject: |
|
|
I have been building a new local server, will be http-replicator server, and so thought that using rsync mirror and http-replicator which are setup on my workstation would be good idea.
When i got to to chroot stage of install i downloaded a copy of my notebooks make.conf as it has same cpu and already had rsync and replicator settings which work fine. I amended USE setting and did emerge sync, all went well.
Thin i did bootstrap, all went well. The i did emerge system and all seemed to be going well with packages being grabbed from http-replicator cache.
I went to bed.
In morning i found that it had stalled on python, it appeared to have downloaded pyhton 3 times at 7mb but each time emerge failed saying python could not be downloaded.
I disabled http-replicator in make.conf and tried again, this time it resumed python download from 3mb and finished it ok with total 7mb. This i thought strange and check cache dir and found that in there python indeed was only 3mb.
So http-replicator found python in cache dir but did not recognise that it was an incomplete download so kept serving it as full file and so emerge kept failing.
any way of preventing this?
I am using ver 2.0-r2
cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Sun Aug 08, 2004 5:15 pm Post subject: |
|
|
I apprears you may be using an old replicator config or have an error in the setup. I still want to check it out though.
You mentioned in your other posts about sharing /usr/portage over nfs. You weren't doing something like that were you?
Could you PM me your make.conf, replicator config and http-replicator logs? Plus any other info like either your emerge.log or a copy of the last output from the emerge?
Last edited by flybynite on Wed Aug 11, 2004 1:11 am; edited 1 time in total |
|
| Back to top |
|
|
meowsqueak
Veteran
Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand
|
| Posted: Sun Aug 08, 2004 10:41 pm Post subject: |
|
|
Oh, is /etc/http-replicator.conf deprecated now? I only just noticed the configuration file is now /etc/conf.d/http-replicator.conf. Does this mean the telnet monitor port is now unsupported?
Last edited by meowsqueak on Sun Aug 08, 2004 10:55 pm; edited 1 time in total |
|
| Back to top |
|
|
meowsqueak
Veteran
Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand
|
| Posted: Sun Aug 08, 2004 10:54 pm Post subject: |
|
|
Looking good! Latest version with proxy support seems to be working well here
How difficult would it be to add bandwidth-limiting on the http-replicator - external proxy transfer? Does http-replicator use a new wget process to request the file from the external proxy, or does it send the HTTP commands directly itself? |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Aug 11, 2004 12:52 am Post subject: |
|
|
| meowsqueak wrote: | | Oh, is /etc/http-replicator.conf deprecated now? I only just noticed the configuration file is now /etc/conf.d/http-replicator.conf. Does this mean the telnet monitor port is now unsupported? |
Yes, the testing version of http-replicator uses /etc/conf.d/http-replicator for the config.
telnet monitor was removed now that the logging is much improved. You can use 'tail -f /var/log/http-replicator.log' from a local terminal or ssh for the same type of monitor. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Aug 11, 2004 1:02 am Post subject: |
|
|
| meowsqueak wrote: | Looking good! Latest version with proxy support seems to be working well here
How difficult would it be to add bandwidth-limiting on the http-replicator - external proxy transfer? Does http-replicator use a new wget process to request the file from the external proxy, or does it send the HTTP commands directly itself? |
New features are planned
Bandwidth limiting wasn't one being considered Nobody ever asked for it before... Http-Replicator is all Python and doesn't use any external programs such as wget. |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Wed Aug 11, 2004 8:25 am Post subject: |
|
|
| flybynite wrote: | I apprears you may be using an old replicator config or have an error in the setup. I still want to check it out though.
You mentioned in your other posts about sharing /usr/portage over nfs. You weren't doing something like that were you?
Could you PM me your make.conf, replicator config and http-replicator logs? Plus any other info like either your emerge.log or a copy of the last output from the emerge? |
Hello, i had tried sharing portage over nfs but it did not work well and it was not being done at the time i have problem, though the file concerned may have been affected whe i had.
Do you want the conf/log files from the server or client or both?
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
Gherald2
Guru
Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA
|
| Posted: Wed Aug 11, 2004 8:59 am Post subject: |
|
|
carpman, if for whatever reason you do decide to go with an NFS share instead of http-replicator you probably only want to share /usr/portage/distfiles, not the entire /usr/portage directory.
Now about your problem, my guess is you haven't set RESUMECOMMAND in you make.conf as outlined in the first post to this thread. But even with the correct one, sometimes downloads fail for strange reasons and you have to manually flush the cache, that is, on the server:
rm /var/cache/http-replicator/<filename>
... where <filename> is the file that failed to download.
_________________
Unregistered Linux User #17598363 |
|
| Back to top |
|
|
JohnHerdy
n00b
Joined: 11 Jul 2002
Posts: 61
|
| Posted: Wed Aug 11, 2004 9:38 am Post subject: |
|
|
| flybynite wrote: | | New features are planned |
Can you tell us more?
I have one feature request; I would like to see in the logfile the IP-address of the proxyclient that is requesting a file. Then it's possible to setup some auditing and prevent abuse. Thanks a lot! |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Wed Aug 11, 2004 10:08 am Post subject: |
|
|
| Gherald wrote: | carpman, if for whatever reason you do decide to go with an NFS share instead of http-replicator you probably only want to share /usr/portage/distfiles, not the entire /usr/portage directory.
Now about your problem, my guess is you haven't set RESUMECOMMAND in you make.conf as outlined in the first post to this thread. But even with the correct one, sometimes downloads fail for strange reasons and you have to manually flush the cache, that is, on the server:
rm /var/cache/http-replicator/<filename>
... where <filename> is the file that failed to download. |
I have stopped using nfs share of /usr/portage
The problem only occured on a new build of gentoo not updates to other machines on network. I did remove offending file from cache.
This is the relevent part of make.conf from client:
| Code: |
PORTAGE_TMPDIR=/var/tmp
PORTDIR_OVERLAY=/usr/local/portage
# GENTOO_MIRRORS="http://www.mirror.ac.uk/sites/www.ibiblio.org/gentoo http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo"
# Default fetch command (5 tries, passive ftp for firewall compatibility)
http_proxy="http://192.168.1.5:8080"
#FETCHCOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
# SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
SYNC="rsync://192.168.1.5/gentoo-portage"
RSYNC_RETRIES="3"
|
As you can see i did have the resume section uncommented.
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
