Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Tue Oct 26, 2004 4:20 pm    Post subject: Reply with quote

JohnHerdy wrote:

Sorry for asking a stupid question, but I couldn't find it on your website; is the latest version available on your website compliant with 2.0.51-clients?


Sorry, the website is out of date, There are two versions out, both are only listed or linked in the howto at the start of this thread.

All versions of http-replicator will work with the latest portage. But, repcacheman will barf with an error on >=portage-2.0.51. repcacheman errors will not affect the operation of http-replicator.

You can download a new repcacheman here:
http://www.updatedlinux.com/replicator/portagefix/repcacheman

You'll need to copy it to /usr/bin/repcacheman and chmod +x

Or, just edit 2 lines on your existing /usr/bin/repcacheman. Change t[0] to t["MD5"]

I'll have the fix in the ebuild soon.....


Last edited by flybynite on Fri Oct 29, 2004 7:53 pm; edited 1 time in total
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Tue Oct 26, 2004 5:58 pm    Post subject: Reply with quote

flybynite wrote:
All versions of http-replicator will work with the latest portage


Hi flybynite,

Unfortunately this is not the case. 100% of the 2.0.51-clients don't work, 100% of the 2.0.50-clients work. The moment I migrate a client to 2.0.51 http-replicator doesn't work anymore. In our configuration we use a proxy-server (with validation). It seems that something is going wrong with the connection to the proxy. Our proxy resolves the ip-address for the mirrors but when I use http-replicator on a 2.0.51-client the proxy always returns unknown host. When I do a emerge-webrsync on a 2.0.51-client the tarball is fetched from the http-replicator server.

Thanks,
John.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Tue Oct 26, 2004 8:34 pm    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

kmarasco wrote:

Is the second syntax ok with http-replicator? I'm using portage portage-2.0.51-r2.


It's going to be a long day :-(

This should fix it...
Code:

rm /usr/portage/packages/app-editors/beaver-0.2.6.tbz2
rm /usr/portage/packages/All/beaver-0.2.6.tbz2


Then try the replicator syntax again. This will help me figure out where the problem is.

Http-replicator doesn't support resuming yet. This syntax disables resuming and forces wget to overwrite the partial file.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Tue Oct 26, 2004 8:54 pm    Post subject: Reply with quote

JohnHerdy wrote:

Unfortunately this is not the case. 100% of the 2.0.51-clients don't work



2.0.51 works for me :-)

I need some output and logs before I can even guess at whats wrong....
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Tue Oct 26, 2004 9:42 pm    Post subject: Reply with quote

Ok did some more research, it seems that the new version of portage doesn't respect my .wgetrc-settings anymore. After adding the following lines to /etc/make.conf 2.0.51-clients work as well;

http_proxy="http://YourMirrorHere.com:8080"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"

With 2.0.50 this wasn't neccesary the proxy-settings where read from .wgetrc. Hmmm, strange...
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: White Salmon, WA

PostPosted: Tue Oct 26, 2004 10:10 pm    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

flybynite wrote:

This should fix it...
Code:

rm /usr/portage/packages/app-editors/beaver-0.2.6.tbz2
rm /usr/portage/packages/All/beaver-0.2.6.tbz2

Then try the replicator syntax again. This will help me figure out where the problem is.


Below code after deleting the files from both the client and the server.
Code:
kristel root # emerge -gK beaver
Fetching binary packages info...
Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
ooo
  -- DONE!

Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--12:29:48--  http://192.168.2.2/portage/packages/All/beaver-0.2.6.tbz2
           => `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:80... connected.
HTTP request sent, awaiting response... 404 Not Found
12:29:48 ERROR 404: Not Found.

Fetcher exited with a failure condition.
!!! CATEGORY info missing from info chunk, aborting...
kristel root #

If I recreate the binary package, I get back to the originally posted error.

Note: I had ethereal observe the traffic and the http-replicator proxy does not appear to get involved with the locally served binaries from apache. I'm a rookie at ethereal, so I may have missed something, but it appears that when merging binaries using the PORTAGE_BINHOST variable in make.conf that the http_proxy variable is ignored (only for emerge -gK. The proxy is definitely used when source packages are pulled.).
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Tue Oct 26, 2004 11:13 pm    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

kmarasco wrote:

If I recreate the binary package, I get back to the originally posted error.


Thanks, that confirms some things for me.

You don't have FETCHCOMMAND set do you?

kmarasco wrote:

Note: I had ethereal observe the traffic and the http-replicator proxy does not appear to get involved with the locally served binaries from apache.


Http-replicator isn't involved at all in this transaction. However RESUMECOMMAND seems to be and that is the problem. I find it "interesting" that portage uses RESUMECOMMAND when there is nothing to resume, I've filed a portage bug to get more info.

I use PORTAGE_BINHOST also, but with an ftp mirror. If you need a workaround, use an FTP server as your BINHOST till I get this worked out.
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: White Salmon, WA

PostPosted: Wed Oct 27, 2004 9:21 am    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

flybynite wrote:

I use PORTAGE_BINHOST also, but with an ftp mirror. If you need a workaround, use an FTP server as your BINHOST till I get this worked out.

I set up vsftp and I have the same issue.
Code:

kristel root # emerge -gK beaver
Fetching binary packages info...
 * No password provided for username 'anonymous'

Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
oooo
  -- DONE!

Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--23:29:42--  ftp://192.168.2.2/All/beaver-0.2.6.tbz2
           => `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /All ... done.
==> PASV ... done.    ==> RETR beaver-0.2.6.tbz2 ... done.
Length: 144,983 (unauthoritative)

100%[=================================================================================>] 144,983      107.91K/s

23:29:43 (107.85 KB/s) - `/usr/portage/packages/All//${FILE}' saved [144983]

!!! CATEGORY info missing from info chunk, aborting...
kristel root #


Hmmm, any more thoughts? Typo/poor attention to detail on my part?
Code:
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
[/quote]
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: White Salmon, WA

PostPosted: Thu Oct 28, 2004 10:24 am    Post subject: Alternative Resume Syntax Reply with quote

Thanks for your help flybynite. Since I experienced the bug with both ftp and http, I tried to find an alternative syntax for the resume command. I think that the following syntax should have the same effect, but does not require that the file name variable be passed in order to replace a partial file. It is as follows:
Code:
RESUMECOMMAND="/usr/bin/wget -N -t 5 --passive-ftp \${URI} -P \${DISTDIR}"

With the -N option, Only new files will be downloaded in the place of the old ones. Using -N, a file is considered new if one of these two conditions are met:

1. A file of that name does not already exist locally.
2. A file of that name does exist, but the remote file was modified more recently than the local file.

More importantly, if the local file does not exist, or the sizes of the files do not match, Wget will download the remote file no matter what the time-stamps say.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Fri Oct 29, 2004 12:42 am    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

kmarasco wrote:
I set up vsftp and I have the same issue.

Typo/poor attention to detail on my part?


Nope, my part, sorry. I was using ftp without error on one of my boxes - because I had changed the RESUMECOMMAND so many times in testing that I left it wrong. The portage bug affects both ftp and http.


kmarasco wrote:

I tried to find an alternative syntax for the resume command. I think that the following syntax should have the same effect, but does not require that the file name variable be passed in order to replace a partial file.


I appreciate the help. Wget interacting with portage and RESUMECOMMAND together had so many options finding the right combination is complex. I wish the wget doc's were better....

The good news is that this problem should go away soon!!

Thanks to you, I reported this bug and the portage dev's have deemed it worthy. It has been fixed in CVS. This means the next release of portage should contain the fix and then you can go back to using $FILE!!

Until then, your -N suggestion is probably the best fix. It's not a perfect solution for http-replicator because of a couple side effects that limit http-replicator. One such side effect is that with -N, an internet connection must exist. With the original RESUMECOMMAND, http-replicator doesn't need an internet connection to serve from the cache.
Back to top
View user's profile Send private message
matbintang
n00b
n00b


Joined: 12 Aug 2003
Posts: 22
Location: Canada

PostPosted: Fri Oct 29, 2004 3:42 am    Post subject: Reply with quote

flybynite!

You Da Man!!

:D
_________________
$>cd /pub
$>more beer
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: White Salmon, WA

PostPosted: Fri Oct 29, 2004 5:09 am    Post subject: Reply with quote

Thanks again for all of your help. I commend you for your excellent work and ongoing support effort :!:
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Fri Oct 29, 2004 8:06 pm    Post subject: Reply with quote

Thanks ( smiles proudly)

I just hope you still feel that way when I say you need to change portage_util.writemsg back to portage.writemsg !!

Or download repcacheman ver 3.2 at http://www.updatedlinux.com/replicator/portagefix/repcacheman
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1045
Location: Gentoo Forums

PostPosted: Sat Oct 30, 2004 10:14 am    Post subject: Reply with quote

i do have a problem: repcacheman runs every 5 minutes but even though it should have nothing to do (i.e. i didn't download any packages and /usr/portage/distfiles/ is empy) it eats a lot of my precious memory (my server is an oldie):
Code:
  PID  PPID  UID USER     RUSER    TTY         TIME+  %CPU %MEM S COMMAND
4174  4163    0 root     root     ?          0:28.75 10.6 12.7 R repcacheman

how could i prevent that?
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Oct 30, 2004 4:24 pm    Post subject: Reply with quote

The release of portage 2.0.51 has caused a number of problems.

These problems haven't affected how well http-replicator itself runs, only with the support script repcacheman.

I have updated ebuilds with changes to support portage 2.0.51. I need help testing:

http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.6a.tar.bz2

This release contains 5 versions:
* net-misc/http-replicator :
[ ] 2.0-r2 (0) OVERLAY
[ ] 2.0-r3 (0) OVERLAY
[ ] 2.1_rc3 (0) OVERLAY
[ ] 2.1_rc3-r1 (0) OVERLAY
[ I] 2.1 (0) OVERLAY


2.0-r2 and 2.1_rc3 DEPEND on portage 2.0.50 and won't work with the current stable portage 2.0.51 I left these in only for the 1 or 2 gentooers who don't like to upgrade :-)

2.0-r3 , 2.1_rc3_r1 and 2.1 will work with any current portage version. 2.1 primarily adds external proxy support and improved logging.

2.1 is the latest and is stable, but the changes aren't fully documented in the howto yet.

Follow the howto at the start of this thread for all versions. Then If you upgraded to 2.1 -look at these changes:

**Note The config file location has changed in 2.1. Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults.

**Note repcacheman has changed in 2.1. If you changed the default cache dir or the default user, you must call repcacheman with those options.
Code:

repcacheman --user USER --dir /path/to/cache


Last edited by flybynite on Sat Oct 30, 2004 9:56 pm; edited 2 times in total
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Oct 30, 2004 4:27 pm    Post subject: Reply with quote

BlinkEye wrote:

how could i prevent that?


Fix is in the above post....
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Oct 30, 2004 4:47 pm    Post subject: Reply with quote

I had a question about repcacheman and what it does and how often it should be run.

Repcacheman only needs to run after emerge is run on the server ( except emerge sync)

Here is some more info on repcacheman:
Quote:


Replicator can't share the same dir with portage because portage doesn't play nice.

This means only on the server, emerged packages exist in both replicator's cache and /distfiles. The files in /distfiles are wasted space and dups. One replicator function is to delete the dups in distfiles.

The other function of repcacheman is to move any new files from /distfiles to replicators cache. The only way to get files in /distfiles that aren't in replicators cache is by ftp, since replicator doesn't support ftp. So if you run emerge on the server and the package isn't on any http mirror portage will fall back to ftp and retrieve the file. Running repcacheman will check the md5 and then move the new file into the cache.

The only time files are ftp'd into /distfiles is after emerge is running on the server.

The only time dups occur is after running emerge on the server.

The only time repcacheman needs to run is after any emerge's on the server.

The best option is to run emerge's on the server like this:
Code:

emerge -uDva world && repcacheman


This will update world including dep's - asking if you won't to continue - and then run repcacheman after the emerge is finished.

Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 01, 2004 4:33 am    Post subject: Reply with quote

I upgraded to 2.1 but I'm getting this problem on one of my LANs. The following spews out about 20 times:

Code:
...
Connecting to 10.16.10.224:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz [following]
--17:30:43--  http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz
           => `/usr/portage/distfiles/fontconfig-2.2.3.tar.gz'
Connecting to 10.16.10.224:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz [following]
...
20 redirections exceeded.
!!! Couldn't download fontconfig-2.2.3.tar.gz. Aborting.


Funnily enough, it works fine on another LAN I look after. I'm using an external proxy too, but that IP address (10.16.10.224) is the address of my http-replicator server, and I'm running 'emerge -Duav world' on the same host, so it's trying to connect to itself and then failing. My conf file includes '--ip 10.*.*.*'
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 01, 2004 8:12 pm    Post subject: Reply with quote

Ok, I did some reading into the 301 response:

http://www.checkupdown.com/status/E301.html

According to the output, wget first tries to download http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz. The proxy request is sent and the response is a 301 error (presumably from the original server?). The Location specifies http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz as the alternative URL, and wget follows this. But when it tries to download this, it gets another 301 with the location set to the exact same URL and the process repeats until the limit (20) is hit. So, is this a problem with this server in particular, or is it a problem with the http-replicator proxy? I suspect the server is fine because a manual wget of the osuosl.org file works fine (no followed location). Does http-replicator cache server responses or something?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Wed Nov 03, 2004 6:56 pm    Post subject: Reply with quote

First, go ahead and update the link to http://gentoo.osuosl.org/ - since http://gentoo.oregonstate.edu/ is outdated anyway. I've updated the howto.

The response isn't cached by http-replicator- might be cached by your ISP or somewhere else.

Then if you still have this problem, upgrade to http-replicator 2.1, set debug in the config and post the logs...
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Wed Nov 03, 2004 8:26 pm    Post subject: Reply with quote

Hmmm, I wonder if my squid proxy is caching something - here's the log:

Code:
04 Nov 2004 09:23:19 DEBUG: HttpClient 170 connected to 10.16.10.224:42290
04 Nov 2004 09:23:19 DEBUG: HttpClient 170 received header:

  GET http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz HTTP/1.0
  User-Agent: Wget/1.9
  Host: gentoo.oregonstate.edu
  Accept: */*

04 Nov 2004 09:23:19 INFO: HttpClient 170 received request for http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz
04 Nov 2004 09:23:19 DEBUG: HttpClient 170 connecting to gentoo.oregonstate.edu
04 Nov 2004 09:23:19 DEBUG: HttpServer 170 connected to 10.10.130.123:3128
04 Nov 2004 09:23:19 DEBUG: HttpServer 170 received header:

  HTTP/1.0 301 Moved Permanently
  Date: Wed, 03 Nov 2004 20:23:19 GMT
  Server: Apache/2.0.52 (Debian GNU/Linux)
  Location: http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz
  Content-Length: 266
  Content-Type: text/html; charset=iso-8859-1
  X-Cache: MISS from squid-proxy
  Proxy-Connection: close

04 Nov 2004 09:23:19 DEBUG: HttpClient 170 closed



Why is the first GET still trying http://gentoo.oregonstate.edu ? I don't have that in make.conf any more. Emerge says

Code:
>>> Downloading http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz


Is http://gentoo.mirrors.pair.com providing me with this perhaps? I tried:

Code:
$ GENTOO_MIRRORS="http://gentoo.osuosl.org" sudo emerge -a --oneshot fontconfig
...
>>> Downloading http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz


Not sure I understand what is going on here...
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Thu Nov 04, 2004 6:56 pm    Post subject: Reply with quote

meowsqueak wrote:
Hmmm, I wonder if my squid proxy is caching something


Squid's a likely culprit, but lets fix the link first......

meowsqueak wrote:

Why is the first GET still trying http://gentoo.oregonstate.edu ? I don't have that in make.conf any more.
Not sure I understand what is going on here...


If you followed the howto, you put gentoo.oregonstate.edu into your /etc/portage/mirrors.

That "local" mirror is downloaded before the make.conf mirrors. I'd bet you just need to change the /etc/portage/mirrors....
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Nov 04, 2004 9:30 pm    Post subject: Reply with quote

flybynite wrote:
That "local" mirror is downloaded before the make.conf mirrors. I'd bet you just need to change the /etc/portage/mirrors....


Bingo! I completely forgot about that file. Thanks for the help, I'm sorted now.
Back to top
View user's profile Send private message
Maxwell
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2003
Posts: 96

PostPosted: Mon Nov 08, 2004 9:33 pm    Post subject: Problems using it Reply with quote

Well, i suppose i must be doing something stupid, but i need some help.

I installed http_replicator as i should, but as i have an http proxy port=8080, i switched replicator's port to 5000. But it looks like http_replicator isn't used by my notebook, in my internal lan. the value "http_proxy" in make.conf is the ip and door of the server, correct?

Both my /usr/portage/distfiles and my /var/log/http-replicator are empty. The log file also doesn't report anything when i emerge a file at the server.

Suggestions?
_________________
Freedom works. Use it!
Linux, by Gentoo
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 08, 2004 9:50 pm    Post subject: Re: Problems using it Reply with quote

Maxwell wrote:
Both my /usr/portage/distfiles and my /var/log/http-replicator are empty. The log file also doesn't report anything when i emerge a file at the server.


/var/cache/http-replicator?

Did you start the http-replicator daemon?

Code:
 # /etc/init.d/http-replicator start
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 22, 23, 24  Next
Page 8 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum