Code: Select all
http://address.of.machine.a/gentoo/mirror/Code: Select all
emerge --fetchonly <package>Code: Select all
FETCHCOMMAND="ssh distfiles@nada 'umask 022; /usr/bin/wget -nc -t 5 --passive-ftp \${URI} -P /home/distfiles/public_html/distfiles/' && echo \${URI} | /bin/sed -r 's/.*\\//http:\\/\\/nada\\/~distfiles\\/distfiles\\//' | /usr/bin/wget -t 5 --passive-ftp -P \${DISTDIR} -i -"
RESUMECOMMAND="ssh distfiles@nada 'umask 022; /usr/bin/wget -c -t 5 --passive-ftp \${URI} -P /home/distfiles/public_html/distfiles/' && echo \${URI} | /bin/sed -r 's/.*\\//http:\\/\\/nada\\/~distfiles\\/distfiles\\//' | /usr/bin/wget -t 5 --passive-ftp -P \${DISTDIR} -i -"
GENTOO_MIRRORS="http://nada/~distfiles <other mirrors>"
Code: Select all
FETCHCOMMAND="sudo -H -u distfiles /usr/bin/wget -t 5 --timeout=2700 http://nada/~distfiles/getdistfile2.php?url=\${URI} -P \${DISTDIR}"
RESUMECOMMAND="sudo -H -u distfiles /usr/bin/wget -c -t 5 --timeout=2700 http://nada/~distfiles/getdistfile2.php?url=\${URI} -P \${DISTDIR}"
GENTOO_MIRRORS="http://nada/~distfiles <other mirrors>"
Code: Select all
<?
$url = $_GET['url'];
if(substr_count($_SERVER["REMOTE_ADDR"], "192.168.1.") != 1)
{
echo "Invalid Remote Address!";
exit;
}
$exec_string = "/usr/local/phpbin/getdistfiles ";
$exec_arg = escapeshellcmd($url);
$output = exec("$exec_string $exec_arg");
header("Location: $output");
?>
Code: Select all
#!/bin/bash
umask 022 &>/dev/null
echo "$1" > /tmp/wgetfile 2>/dev/null
/usr/bin/wget -nc -c -t 5 --passive-ftp -i /tmp/wgetfile -P /home/distfiles/public_html/distfiles &>/dev/null
/bin/rm /tmp/wgetfile 2>/dev/null
echo $1 | /bin/sed -r 's/.*\//http:\/\/nada\/~distfiles\/distfiles\//'
Thank youCode: Select all
FETCHCOMMAND="ssh distfiles@nada 'umask 022; /usr/bin/wget -nc -t 5 --passive-ftp \${URI} -P /home/distfiles/public_html/distfiles/' && echo \${URI} | /bin/sed -r 's/.*\\//http:\\/\\/nada\\/~distfiles\\/distfiles\\//' | /usr/bin/wget -t 5 --passive-ftp -P \${DISTDIR} -i -" RESUMECOMMAND="ssh distfiles@nada 'umask 022; /usr/bin/wget -c -t 5 --passive-ftp \${URI} -P /home/distfiles/public_html/distfiles/' && echo \${URI} | /bin/sed -r 's/.*\\//http:\\/\\/nada\\/~distfiles\\/distfiles\\//' | /usr/bin/wget -t 5 --passive-ftp -P \${DISTDIR} -i -" GENTOO_MIRRORS="http://nada/~distfiles <other mirrors>"
I have never used prozilla, but as long as it has a comparable command line input options (and does not require interactive user attention) converting the settings should be trivial.Nar wrote:Hi
Is it posible to do it with prozilla ?
Thank you
Nar
Why bother with a proxy? Just share /usr/portage/distfiles ! Portage will handle everything else for you. That should take a couple of seconds to set up. Also, if your build system is configured for binary compatibility across your machines, have emerge build binaries (using cache and distcc) and share the binary directory too. You could even make a script that builds the package, and then uses ssh to emerge the pre-built binary on every machine.PopeKetric wrote:Forgive me if this has been asked before, but is there a 'documented' way of setting up a machine to proxy the distfiles for portage? I mantain several Gentoo boxes around here and would like to make one machine do the following:
Act as kind of a mirror for the portage tree. Only grabbing files when they are new, and requested by one of my local machines. So machine B asks machine a for file xyz.tbz which then downloads xyz.tbz. So when machine C asks for xyz.tbz it just grabs the file from machine A directly, without using my horribly busy net connection.
--Pope
Ever look at how Intermezzo works? Its a whole lot of pain, and definately not recommended, especially for something like this. As for being able to emerge sync or emerge distfiles ... you would still have a clash if you did two syncs or downloaded two of the same distfiles at the same time. In fact, you'll get lots of issues when updating tthe same files on different machines. If you want everything synced at once, then you don't want multiple machines to touch the tree at once!!marty wrote:You could possibly use one of the distributed replicating/synchronizing filesystems such
as coda or intermezzo to accomplish this. I haven't
done this myself yet, so this is all in theory. The
benefit would be that you would have no "server".
All the participants in the intermezzo/coda "cluster"
would be able to "emerge sync" or download new
distfiles by emerging ebuilds which would then become
available to the other machines as needed.
Code: Select all
#!/bin/sh
#
# Wraps fetch utility as called by portage, so that downloaded files can be
# subsequently uploaded to a local mirror.
#
# Can only deal with one file at a time, but (I assume) this is all portage
# needs.
#
# Specify the command and arguments used to fetch the requested file
# FETCH_ARGS must be constructed so that when invoked as:
#
# $FETCH_CMD $FETCH_ARGS $DEST $URI
#
# the file at $URI is saved into the file $DEST. If we are aksed to
# resume a download, the invoked command will be:
#
# $FETCH_CMD $FETCH_RESUME $FETCH_ARGS $DEST $URI
#
FETCH_CMD="/usr/bin/wget"
FETCH_ARGS="-t 5 --passive-ftp -O "
FETCH_RESUME="-c"
# Files will not be uploaded if $URI matches this regexp
EXCLUDE_FILES_MATCHING="whitley"
# The command to use to upload the file, and the destination for the file
# This will be invoked as:
#
# $UPLOAD_CMD $DEST $UPLOAD_DEST
#
# where $DEST points to the downloaded file.
UPLOAD_CMD="scp -B -i /root/.ssh/gentoo_id_rsa"
UPLOAD_DEST="gentoo@gentoo.whitley.unimelb.edu.au:pub/distfiles/."
#### CONFIGURATION OPTIONS END HERE ####
USAGE="$0 [-c] <URI> <DESTINATION>"
# Sanity check arguments
if [ $# -eq 3 ]
then
if [ $1 != "-c" ]
then
echo "$USAGE";
exit 1;
fi
URI="$2"
DEST="$3"
else
if [ $# -ne 2 ]
then
echo "$USAGE";
exit 1;
fi
URI="$1"
DEST="$2"
fi
## Do the actual download, resuming if necessary
if [ $# -eq 3 ]
then
$FETCH_CMD $FETCH_RESUME $FETCH_ARGS $DEST $URI
else
$FETCH_CMD $FETCH_ARGS $DEST $URI
fi
## Upload the file, if successful and if it is not to be excluded
if [ $? -eq 0 ]
then
FNAME_TEST=`echo "$URI" | grep "$EXCLUDE_FILES_MATCHING"`
if [ "x$FNAME_TEST" == "x" ]
then
echo "Uploading to local mirror"
$UPLOAD_CMD $DEST $UPLOAD_DEST
fi
fi
Code: Select all
FETCHCOMMAND="/usr/local/bin/portage-get \${URI} \${DISTDIR}/\${FILE}"
RESUMECOMMAND="/usr/local/bin/portage-get -c \${URI} \${DISTDIR}/\${FILE}"
How does one create the ssh keys for scp ?Code: Select all
... # where $DEST points to the downloaded file. UPLOAD_CMD="scp -B -i /root/.ssh/gentoo_id_rsa" ...
ssh-keygen.grudge wrote:How does one create the ssh keys for scp ?
:?:
Code: Select all
ssh-keygen -t rsaCode: Select all
ssh remote.machine.net cat \>\> .ssh/authorized_keys < .ssh/id_rsa.pubCode: Select all
--- /usr/bin/http-replicator 2005-04-22 14:14:20.000000000 -0400
+++ /usr/bin/http-replicator 2005-07-01 17:19:10.000000000 -0400
@@ -7,6 +7,12 @@
import asyncore, socket, os, time, calendar, sys, re, optparse, logging
+#
+class NotPermitedURL (Exception):
+ pass
+
# LISTENER
#
# Class [Listener] is a subclass of [asyncore.dispatcher].
@@ -103,6 +109,7 @@
200: 'OK',
206: 'Partial Content',
304: 'Not Modified',
+ 401: 'Unauthorized',
404: 'Not Found',
416: 'Requested Range Not Satisfiable',
503: 'Service Unavailable' }
@@ -272,6 +279,7 @@
self.direct = 'http://' + body['host'] + head[1].rstrip('/') # save full url for file listings
else:
self.direct = '.' # fall back on relative url is host is not sent
+
self.counterpart = HttpServer(counterpart=self) # create counterpart
@@ -280,6 +288,29 @@
if not response: # no server response prescribed; connect
try:
+ # <---------------------------------------------->
+ # verifying URLs
+ # <---------------------------------------------->
+ import portage
+ test = portage.settings["GENTOO_MIRRORS"].split()
+ f = open('/etc/http-replicator/allows','r')
+ line = f.readlines()
+ self.log.error('Test : %s' % head[1])
+ f.close()
+ test = [l + 'distfiles' for l in test]
+ test = test + line
+ self.log.error('Test : %s' % test)
+ for l in test:
+ desturl = self.spliturl(l)
+ if desturl:
+ desthost, destport, destpath = desturl.groups() # parse url
+
+ if desthost.index(host) != -1 and destpath.index(path[:path.rindex('/')]) != -1:
+ break
+
+ else:
+ raise NotPermitedURL("Url not permited")
+ # <---------------------------------------------->
assert proxy, 'direct request to hidden or unaccessible file'
self.log.debug('connecting to %s', host)
self.counterpart.create_socket(socket.AF_INET, socket.SOCK_STREAM) # create socket
@@ -289,11 +320,18 @@
else: # forward request to the server
assert port == 80 or port > 1024, 'illegal attempt to connect to port %i' % port # security
self.counterpart.connect((host, port)) # connect to server
+ except NotPermitedURL: # connection failed
+ exception, value, traceback = sys.exc_info() # get exception info
+ self.log.error('connection failed: %s', value) # log reason of failure
+ print >> self.counterpart.data, '<html><body><pre><h1>You are not authorized to access to this URL</h1></pre></body></html>'
+ response = 503 # service unavailable
+ self.log.error('connection failed: %s', host)
except: # connection failed
exception, value, traceback = sys.exc_info() # get exception info
self.log.error('connection failed: %s', value) # log reason of failure
print >> self.counterpart.data, '<html><body><pre><h1>Service unavailable</h1></pre></body></html>'
response = 503 # service unavailable
+ self.log.error('connection failed: %s', host)
self.counterpart.set_header(head, body) # reconstruct header and set counterpart.sending to true
Code: Select all
meg ~ # patch -p0 < http-replicator.patch
patching file /usr/bin/http-replicator
patch: **** malformed patch at line 14: @@ -103,6 +109,7 @@