Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Aug 05, 2004 5:47 am    Post subject: You asked for External proxy support!! Reply with quote

New version ready for testing!! Although the init code,logging, and options are cleaned up, this version primarily adds external proxy support.

Current users not needing external proxy support should wait for the next stable release. Experienced users can help test.

If your don't have any version of http-replicator installed and wish to help test, follow the HOWTO first, then upgrade to this latest version.

1. Download and install the ebuild in the portage overlay directory (/usr/local/portage by default), then emerge http-replicator.

Download:

http-replicator-flybynite-1.5.tar.bz2

which contains both the old and new ebuild's.

Code:

# cd /usr/local/portage
# tar -xvjf /PathToFile/http-replicator-flybynite-1.5.tar.bz2


Then emerge:
Code:

# emerge -u http-replicator


This will upgrade http-replicator.

Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults.

**Note repcacheman has changed. If you changed the default cache dir or the default user, you must call repcacheman with those options.
Code:

repcacheman --user USER --dir /path/to/cache


2. You can downgrade http-replicator if you have any problems with the new version
Code:

# emerge  =net-misc/http-replicator-2.0-r2


Please report your experience!!


[Edit] update to rc3


Last edited by flybynite on Fri Aug 06, 2004 7:32 pm; edited 2 times in total
Back to top
View user's profile Send private message
Boworr
n00b
n00b


Joined: 30 Jul 2004
Posts: 45
Location: England. Where the history comes from.

PostPosted: Thu Aug 05, 2004 3:06 pm    Post subject: Manually adding items to the cache? Reply with quote

Hi,

Firstly, this is great and saves me a huge amount of time/bandwidth at home.

Is it possible to manually add items to the cache? For example, when I'm at work I'd like to use the companies bandwidth :-) to fetch all my packages, then when I get home copy them over the cache server so the other machines can get them quickly.

Is this possible? If so, what do I copy (where does emerge store temp downloads anyway?) and where do I copy it to?

Thanks
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Aug 05, 2004 3:40 pm    Post subject: Reply with quote

Yes, you can manually add files to the cache. The included script repcacheman will do it for your also.

The safest method would be to copy the files to /usr/portage/distfiles and then run repcacheman. repcacheman is a script designed to import files that were downloaded from ftp or manually added. The script will verify the md5sum of all files before adding them to replicators cache.

The fastest method would be to just copy the files directly to replicators cache which is /var/cache/http-replicator unless you changed the defaults. The only drawback to this method is that the md5sum's aren't checked. If any files your downloaded had errors in them you will have to manually delete them from the cache.

The latest version of replicator will md5sum replicators cache, it's just still in testing.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Aug 05, 2004 7:52 pm    Post subject: Re: You asked for External proxy support!! Reply with quote

flybynite wrote:
New version ready for testing!!


Excellent! I will definitely test this out and let you know how it goes. Thanks!
Back to top
View user's profile Send private message
Boworr
n00b
n00b


Joined: 30 Jul 2004
Posts: 45
Location: England. Where the history comes from.

PostPosted: Thu Aug 05, 2004 9:38 pm    Post subject: Reply with quote

flybynite wrote:
Yes, you can manually add files to the cache. The included script repcacheman will do it for your also.


So where do I copy them from? :P i.e, where does emerge store files with I --fetch them?

Boworr
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Aug 05, 2004 10:39 pm    Post subject: Reply with quote

Emerge stores all the files in /usr/portage/distfiles

Copy them from there to the /usr/portage/distfiles on the box with replicator and then run repcacheman
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Fri Aug 06, 2004 12:14 am    Post subject: Reply with quote

Hello, ok trying to save some disk space on my note book and decrease bandwidth for emerge sync etc i setup rsync mirror and http-replicator... very nice but i have some question as i now plan to build a small local server to be the main rsync/http-replicator.

1. getting rid of /usr/portage on clients, currently i have nfs exported the /usr/portage from server to client... is this good idea? would love to get rid of /usr/portage dir on local machines and just have /usr/portage/http-replocator cache on server

2. disk space and dir partitions for server - with the server i plan to have a seperate partition for portage files, from what i can gather /usr/portage is not that important or large, with http-replicator and the reprocache on cron moving the files to http-replicator cache dir, and it is this cache dir that i can give its own partition.


Am i correct?

cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Fri Aug 06, 2004 12:46 am    Post subject: Reply with quote

umm, seems that nfs exporting /usr/portage to client does not work :(

rsync mirror fails and emege -uD world just seems to go on for ever finding dependancies.


h
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Fri Aug 06, 2004 4:11 am    Post subject: Reply with quote

/usr/portage should be stored somewhere FAST and accessing it over NFS is relatively slow. Every time Portage builds a dependency tree, it has to read lots of files from here.
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Fri Aug 06, 2004 8:11 am    Post subject: Re: You asked for External proxy support!! Reply with quote

flybynite wrote:
New version ready for testing!! Although the init code,logging, and options are cleaned up, this version primarily adds external proxy support.


yahoo!!!, I will test it and report back, thanks a lot!!!
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Fri Aug 06, 2004 12:15 pm    Post subject: Reply with quote

I'm testing the proxy support. Can you tell me where to add my username and password?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Aug 06, 2004 7:48 pm    Post subject: Reply with quote

New http-replicator-2.1_rc3 Includes support for basic proxy authentication (as requested by JohnHerdy)


Everybody testing please update to the latest version and test. You will need to download http-replicator-flybynite-1.5.tar.bz2 Instructions are in this post:
https://forums.gentoo.org/viewtopic.php?t=173226&start=76
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Fri Aug 06, 2004 8:49 pm    Post subject: Reply with quote

flybynite wrote:
New http-replicator-2.1_rc3 Includes support for basic proxy authentication (as requested by JohnHerdy)


Man you have no idea how happy you have made me. It's working like a charm!!!

Minor improvement;
- if the cachedir doesn't exists when the daemon starts create it and set the correct permissions or else the daemon crashes.
- in the default /etc/conf.d/http-replicator add an " at the end of the --external examples, because without the " at the end http-replicator crashes.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Aug 06, 2004 9:25 pm    Post subject: Reply with quote

Your welcome!!

JohnHerdy wrote:

Minor improvement;
- if the cachedir doesn't exist when the daemon starts create it and set the correct permissions or else the daemon crashes.


Actually, replicator will print 'invalid directory' or 'no read/write permission for directory' and then exit :-)

You might have missed it because the instructions are only in the complete HOWTO at the start of this thread, but...

I wrote the repcacheman utility to install http-replicator. It will install and maintain replicators cache. It creates the cache dir if it doesn't exist and then transfers files from the host to the cache.

If your server box is gentoo, you should still run repcacheman to prime the cache with your existing files.

In this release candidate version, if you changed the cache dir from the default /var/cache/http-replicator, or changed the user from portage, you need to set the cache dir and/or the user on the command line like so
Code:

# /usr/bin/repcacheman  --dir /var/cache/http-replicator  --user portage


JohnHerdy wrote:

- in the default /etc/conf.d/http-replicator add an " at the end of the --external examples, because without the " at the end http-replicator crashes.


Thanks, my typo... Fixed...
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Sun Aug 08, 2004 8:25 am    Post subject: Reply with quote

I have been building a new local server, will be http-replicator server, and so thought that using rsync mirror and http-replicator which are setup on my workstation would be good idea.

When i got to to chroot stage of install i downloaded a copy of my notebooks make.conf as it has same cpu and already had rsync and replicator settings which work fine. I amended USE setting and did emerge sync, all went well.

Thin i did bootstrap, all went well. The i did emerge system and all seemed to be going well with packages being grabbed from http-replicator cache.

I went to bed.

In morning i found that it had stalled on python, it appeared to have downloaded pyhton 3 times at 7mb but each time emerge failed saying python could not be downloaded.

I disabled http-replicator in make.conf and tried again, this time it resumed python download from 3mb and finished it ok with total 7mb. This i thought strange and check cache dir and found that in there python indeed was only 3mb.

So http-replicator found python in cache dir but did not recognise that it was an incomplete download so kept serving it as full file and so emerge kept failing.

any way of preventing this?

I am using ver 2.0-r2


cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Aug 08, 2004 5:15 pm    Post subject: Reply with quote

I apprears you may be using an old replicator config or have an error in the setup. I still want to check it out though.

You mentioned in your other posts about sharing /usr/portage over nfs. You weren't doing something like that were you?

Could you PM me your make.conf, replicator config and http-replicator logs? Plus any other info like either your emerge.log or a copy of the last output from the emerge?


Last edited by flybynite on Wed Aug 11, 2004 1:11 am; edited 1 time in total
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Sun Aug 08, 2004 10:41 pm    Post subject: Reply with quote

Oh, is /etc/http-replicator.conf deprecated now? I only just noticed the configuration file is now /etc/conf.d/http-replicator.conf. Does this mean the telnet monitor port is now unsupported?

Last edited by meowsqueak on Sun Aug 08, 2004 10:55 pm; edited 1 time in total
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Sun Aug 08, 2004 10:54 pm    Post subject: Reply with quote

Looking good! Latest version with proxy support seems to be working well here :)

How difficult would it be to add bandwidth-limiting on the http-replicator - external proxy transfer? Does http-replicator use a new wget process to request the file from the external proxy, or does it send the HTTP commands directly itself?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Aug 11, 2004 12:52 am    Post subject: Reply with quote

meowsqueak wrote:
Oh, is /etc/http-replicator.conf deprecated now? I only just noticed the configuration file is now /etc/conf.d/http-replicator.conf. Does this mean the telnet monitor port is now unsupported?



Yes, the testing version of http-replicator uses /etc/conf.d/http-replicator for the config.

telnet monitor was removed now that the logging is much improved. You can use 'tail -f /var/log/http-replicator.log' from a local terminal or ssh for the same type of monitor.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Aug 11, 2004 1:02 am    Post subject: Reply with quote

meowsqueak wrote:
Looking good! Latest version with proxy support seems to be working well here :)

How difficult would it be to add bandwidth-limiting on the http-replicator - external proxy transfer? Does http-replicator use a new wget process to request the file from the external proxy, or does it send the HTTP commands directly itself?



New features are planned :-)

Bandwidth limiting wasn't one being considered :-( Nobody ever asked for it before... Http-Replicator is all Python and doesn't use any external programs such as wget.
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Wed Aug 11, 2004 8:25 am    Post subject: Reply with quote

flybynite wrote:
I apprears you may be using an old replicator config or have an error in the setup. I still want to check it out though.

You mentioned in your other posts about sharing /usr/portage over nfs. You weren't doing something like that were you?

Could you PM me your make.conf, replicator config and http-replicator logs? Plus any other info like either your emerge.log or a copy of the last output from the emerge?



Hello, i had tried sharing portage over nfs but it did not work well and it was not being done at the time i have problem, though the file concerned may have been affected whe i had.

Do you want the conf/log files from the server or client or both?
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Wed Aug 11, 2004 8:59 am    Post subject: Reply with quote

carpman, if for whatever reason you do decide to go with an NFS share instead of http-replicator you probably only want to share /usr/portage/distfiles, not the entire /usr/portage directory.

Now about your problem, my guess is you haven't set RESUMECOMMAND in you make.conf as outlined in the first post to this thread. But even with the correct one, sometimes downloads fail for strange reasons and you have to manually flush the cache, that is, on the server:

rm /var/cache/http-replicator/<filename>

... where <filename> is the file that failed to download.
_________________
Unregistered Linux User #17598363
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Wed Aug 11, 2004 9:38 am    Post subject: Reply with quote

flybynite wrote:
New features are planned :-)


Can you tell us more?

I have one feature request; I would like to see in the logfile the IP-address of the proxyclient that is requesting a file. Then it's possible to setup some auditing and prevent abuse. Thanks a lot!
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Wed Aug 11, 2004 10:08 am    Post subject: Reply with quote

Gherald wrote:
carpman, if for whatever reason you do decide to go with an NFS share instead of http-replicator you probably only want to share /usr/portage/distfiles, not the entire /usr/portage directory.

Now about your problem, my guess is you haven't set RESUMECOMMAND in you make.conf as outlined in the first post to this thread. But even with the correct one, sometimes downloads fail for strange reasons and you have to manually flush the cache, that is, on the server:

rm /var/cache/http-replicator/<filename>

... where <filename> is the file that failed to download.


I have stopped using nfs share of /usr/portage

The problem only occured on a new build of gentoo not updates to other machines on network. I did remove offending file from cache.

This is the relevent part of make.conf from client:

Code:

PORTAGE_TMPDIR=/var/tmp
PORTDIR_OVERLAY=/usr/local/portage
# GENTOO_MIRRORS="http://www.mirror.ac.uk/sites/www.ibiblio.org/gentoo http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo"

# Default fetch command (5 tries, passive ftp for firewall compatibility)
http_proxy="http://192.168.1.5:8080"
#FETCHCOMMAND="/usr/bin/wget -t 5  --passive-ftp \${URI} -P \${DISTDIR}"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp  \${URI} -O \${DISTDIR}/\${FILE}"

# SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
SYNC="rsync://192.168.1.5/gentoo-portage"
RSYNC_RETRIES="3"


As you can see i did have the resume section uncommented.
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Wed Aug 11, 2004 4:49 pm    Post subject: Reply with quote

Like I said, strange things happen even with the correct resume command. I have had such problems from time to time, the solution is to delete the file from the cache and start over.

The way to be safest when doing a long emerge is to fork an emerge -f process:

Open up a second console and add -f to your regular emerge command. Once the first file is downloaded, go back to the first terminal and run your emerge w/o -f.

The downloads will complete sooner so you can make sure everything is fetched ok for whatever long build you are doing.
_________________
Unregistered Linux User #17598363
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5 ... 22, 23, 24  Next
Page 4 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum