Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
wget optimizations
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Mon Jul 29, 2002 5:52 am    Post subject: wget optimizations Reply with quote

Do you have a large bandwidth like me?
If you do:
1) # vim /etc/make.conf
2) add this line:
FETCHCOMMAND='/usr/bin/wget --limit-rate=1000k ${URI} -P ${DISTDIR}'
3) replace the 1000k (--limit-rate=1000k) with your possible value


My results:
[code]
overseas voicu # emerge -f evolution
Calculating dependencies ...done!
>>> emerge dev-util/indent-2.2.6 to /
>>> emerge gnome-base/oaf-0.6.8-r1 to /
>>> emerge dev-util/guile-1.4-r3 to /
>>> emerge gnome-base/gconf-1.0.8-r3 to /
>>> emerge gnome-base/gnome-vfs-1.0.5-r2 to /
>>> Resuming download...

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-vfs-1.0.5.tar.gz
--08:57:14-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-vfs-1.0.5.tar.gz
=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]

100%[======================================================>] 1,048,954 11.47M/s ETA 00:00

08:57:14 (11.47 MB/s) - `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz' saved [1048954/1048954]

>>> emerge gnome-base/gnome-print-0.35-r3 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-print-0.35.tar.gz
--08:57:15-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-print-0.35.tar.gz
=> `/usr/portage/distfiles/gnome-print-0.35.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 991,657 [application/x-tar]

100%[======================================================>] 991,657 860.05K/s ETA 00:00

08:57:16 (860.05 KB/s) - `/usr/portage/distfiles/gnome-print-0.35.tar.gz' saved [991657/991657]

>>> emerge gnome-base/libglade-0.17-r6 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libglade-0.17.tar.gz
--08:57:16-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libglade-0.17.tar.gz => `/usr/portage/distfiles/libglade-0.17.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 418,921 [application/x-tar]
100%[======================================================>] 418,921 845.25K/s ETA 00:00

08:57:17 (845.25 KB/s) - `/usr/portage/distfiles/libglade-0.17.tar.gz' saved [418921/418921]

>>> emerge gnome-extra/gal-0.19.3 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gal-0.19.3.tar.bz2--08:57:17-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gal-0.19.3.tar.bz2
=> `/usr/portage/distfiles/gal-0.19.3.tar.bz2'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
08:57:17 ERROR 404: Not Found.


>>> Downloading ftp://ftp.gnome.org/pub/GNOME/unstable/sources/gal/gal-0.19.3.tar.bz2
--08:57:17-- ftp://ftp.gnome.org/pub/GNOME/unstable/sources/gal/gal-0.19.3.tar.bz2
=> `/usr/portage/distfiles/gal-0.19.3.tar.bz2'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/plain]

[ <=> ] 903,527 106.69K/s

08:57:29 (106.69 KB/s) - `/usr/portage/distfiles/gal-0.19.3.tar.bz2' saved [903527]

>>> emerge gnome-base/control-center-1.4.0.5-r1 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/control-center-1.4.0.5.tar.bz2
--08:57:29-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/control-center-1.4.0.5.tar.bz2
=> `/usr/portage/distfiles/control-center-1.4.0.5.tar.bz2'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 3,382,798 [application/x-tar]

100%[======================================================>] 3,382,798 851.20K/s ETA 00:00

08:57:33 (851.20 KB/s) - `/usr/portage/distfiles/control-center-1.4.0.5.tar.bz2' saved [3382798/3382798]
>>> emerge gnome-base/libghttp-1.0.9-r2 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libghttp-1.0.9.tar.gz
--08:57:33-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/libghttp-1.0.9.tar.gz
=> `/usr/portage/distfiles/libghttp-1.0.9.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 147,094 [application/x-tar]

100%[======================================================>] 147,094 886.71K/s ETA 00:00

08:57:33 (886.71 KB/s) - `/usr/portage/distfiles/libghttp-1.0.9.tar.gz' saved [147094/147094]

>>> emerge gnome-base/bonobo-1.0.19-r1 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-1.0.19.tar.gz
--08:57:33-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-1.0.19.tar.gz => `/usr/portage/distfiles/bonobo-1.0.19.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1,556,542 [application/x-tar]

100%[======================================================>] 1,556,542 840.28K/s ETA 00:00

08:57:35 (840.28 KB/s) - `/usr/portage/distfiles/bonobo-1.0.19.tar.gz' saved [1556542/1556542]

>>> emerge gnome-extra/gtkhtml-1.0.2-r2 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gtkhtml-1.0.2.tar.gz
--08:57:35-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gtkhtml-1.0.2.tar.gz => `/usr/portage/distfiles/gtkhtml-1.0.2.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1,303,882 [application/x-tar]

100%[======================================================>] 1,303,882 89.99K/s ETA 00:00

08:57:50 (89.99 KB/s) - `/usr/portage/distfiles/gtkhtml-1.0.2.tar.gz' saved [1303882/1303882]

>>> emerge app-text/gnome-spell-0.4.1-r2 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-spell-0.4.1.tar.gz
--08:57:50-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gnome-spell-0.4.1.tar.gz
=> `/usr/portage/distfiles/gnome-spell-0.4.1.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 185,220 [application/x-tar]

100%[======================================================>] 185,220 918.17K/s ETA 00:00

08:57:50 (918.17 KB/s) - `/usr/portage/distfiles/gnome-spell-0.4.1.tar.gz' saved [185220/185220]

>>> emerge gnome-extra/bonobo-conf-0.14 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-conf-0.14.tar.gz
--08:57:50-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/bonobo-conf-0.14.tar.gz
=> `/usr/portage/distfiles/bonobo-conf-0.14.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 438,259 [application/x-tar]

100%[======================================================>] 438,259 839.19K/s ETA 00:00

08:57:51 (839.19 KB/s) - `/usr/portage/distfiles/bonobo-conf-0.14.tar.gz' saved [438259/438259]

>>> emerge dev-util/gob-1.0.12 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gob-1.0.12.tar.gz
--08:57:51-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/gob-1.0.12.tar.gz
=> `/usr/portage/distfiles/gob-1.0.12.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 171,225 [application/x-tar]

100%[======================================================>] 171,225 950.07K/s ETA 00:00

08:57:51 (950.07 KB/s) - `/usr/portage/distfiles/gob-1.0.12.tar.gz' saved [171225/171225]

>>> emerge net-mail/evolution-1.0.8-r1 to /

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/evolution-1.0.8.tar.gz
--08:57:51-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/evolution-1.0.8.tar.gz
=> `/usr/portage/distfiles/evolution-1.0.8.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 13,929,927 [application/x-tar]

100%[======================================================>] 13,929,927 853.09K/s ETA 00:00

08:58:07 (853.09 KB/s) - `/usr/portage/distfiles/evolution-1.0.8.tar.gz' saved [13929927/13929927]

>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/db-3.1.17.tar.gz
--08:58:07-- http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/db-3.1.17.tar.gz
=> `/usr/portage/distfiles/db-3.1.17.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1,954,117 [application/x-tar]

100%[======================================================>] 1,954,117 872.57K/s ETA 00:00

08:58:10 (872.57 KB/s) - `/usr/portage/distfiles/db-3.1.17.tar.gz' saved [1954117/1954117]
_________________
All for one and one for All
--

MACPRO machine...


Last edited by DArtagnan on Mon Jul 29, 2002 6:54 am; edited 2 times in total
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Mon Jul 29, 2002 5:55 am    Post subject: Reply with quote

...how did it perform without the rate limiting?
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Mon Jul 29, 2002 6:49 am    Post subject: Reply with quote

delta407 wrote:
...how did it perform without the rate limiting?


Have no idea but if you do not specify it, the app. doesn't use all your band...not like prozilla :-)
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
bazik
Retired Dev
Retired Dev


Joined: 22 Jul 2002
Posts: 277
Location: Behind you.

PostPosted: Mon Jul 29, 2002 7:03 am    Post subject: Reply with quote

DArtagnan wrote:

=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]

100%[======================================================>] 1,048,954 11.47M/s ETA 00:00



I get over 20M/s when I cache all the files in my Proxy first :)
_________________
Gentoo Linux/Sparc Developer
http://dev.gentoo.org/~bazik/
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Mon Jul 29, 2002 7:29 am    Post subject: Reply with quote

bAZiK wrote:
DArtagnan wrote:

=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]

100%[======================================================>] 1,048,954 11.47M/s ETA 00:00



I get over 20M/s when I cache all the files in my Proxy first :)


How do you do this?
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
debian
n00b
n00b


Joined: 24 Jun 2002
Posts: 22

PostPosted: Mon Jul 29, 2002 1:31 pm    Post subject: Reply with quote

DArtagnan wrote:
delta407 wrote:
...how did it perform without the rate limiting?


Have no idea but if you do not specify it, the app. doesn't use all your band...not like prozilla :-)

While you mentioned prozilla...I could get much better downloading speed with prozilla by uncomment the following line in /etc/make.conf:
Code:
FETCHCOMMAND='/usr/bin/proz --no-getch -s ${URI} -P ${DISTDIR}'

I compare it against your method and prozilla is about twice as fast. However, I'm only using ADSL, which may be different in your case. :)

Note: you must `emerge prozilla` first.
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Mon Jul 29, 2002 4:01 pm    Post subject: Reply with quote

debian wrote:
DArtagnan wrote:
delta407 wrote:
...how did it perform without the rate limiting?


Have no idea but if you do not specify it, the app. doesn't use all your band...not like prozilla :-)

While you mentioned prozilla...I could get much better downloading speed with prozilla by uncomment the following line in /etc/make.conf:
Code:
FETCHCOMMAND='/usr/bin/proz --no-getch -s ${URI} -P ${DISTDIR}'

I compare it against your method and prozilla is about twice as fast. However, I'm only using ADSL, which may be different in your case. :)

Note: you must `emerge prozilla` first.


I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass?
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Mon Jul 29, 2002 8:33 pm    Post subject: Reply with quote

bAZiK wrote:
DArtagnan wrote:

=> `/usr/portage/distfiles/gnome-vfs-1.0.5.tar.gz'
Resolving my.proxy... done.
Connecting to my.proxy[my.ip]:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 1,048,954 (12,031 to go) [application/x-tar]

100%[======================================================>] 1,048,954 11.47M/s ETA 00:00



I get over 20M/s when I cache all the files in my Proxy first :)


No, see, if you'll notice it had 12,031 bytes to transfer, so that number is really far off.
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
debian
n00b
n00b


Joined: 24 Jun 2002
Posts: 22

PostPosted: Tue Jul 30, 2002 6:58 am    Post subject: Reply with quote

DArtagnan wrote:
I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass?

Strange, prozilla should work where wget work. Any special restriction imposed by your proxy?
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Tue Jul 30, 2002 7:01 am    Post subject: Reply with quote

debian wrote:
DArtagnan wrote:
I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass?

Strange, prozilla should work where wget work. Any special restriction imposed by your proxy?


I really don't know...what should I ask my admin about it?
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
debian
n00b
n00b


Joined: 24 Jun 2002
Posts: 22

PostPosted: Tue Jul 30, 2002 7:17 am    Post subject: Reply with quote

DArtagnan wrote:
debian wrote:
DArtagnan wrote:
I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass?

Strange, prozilla should work where wget work. Any special restriction imposed by your proxy?


I really don't know...what should I ask my admin about it?

You don't need to ask, what information would you expect from those arrogant bastards(well, I'm an admin but I'm an exception :) ).

First of all we need to know what kind of problem you got when using prozilla, then we might get some hint why wget works where prozilla doesn't.
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Tue Jul 30, 2002 7:20 am    Post subject: Reply with quote

debian wrote:
DArtagnan wrote:
debian wrote:
DArtagnan wrote:
I KNOW!!! Prozilla rocks but it doesn't work for me, I think my prozy blocks my prozilla , is it possible to configure prozilla for proxy pass?

Strange, prozilla should work where wget work. Any special restriction imposed by your proxy?


I really don't know...what should I ask my admin about it?

You don't need to ask, what information would you expect from those arrogant bastards(well, I'm an admin but I'm an exception :) ).

First of all we need to know what kind of problem you got when using prozilla, then we might get some hint why wget works where prozilla doesn't.


Well, don't forget that "wget" allows you to configure it for proxy at: /etc/wget/wget
There you can specify if to use proxy and then address:port
Prozilla does not support proxy configuration...
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
debian
n00b
n00b


Joined: 24 Jun 2002
Posts: 22

PostPosted: Tue Jul 30, 2002 8:30 am    Post subject: Reply with quote

DArtagnan wrote:
Well, don't forget that "wget" allows you to configure it for proxy at: /etc/wget/wget
There you can specify if to use proxy and then address:port
Prozilla does not support proxy configuration...

You can set it up in the shell before executing prozilla:
Code:
export http_proxy="http://yourproxy.com:8080/"

Also, for ftp:
Code:
export ftp_proxy="http://yourproxy.com:8080/"

Btw, you can catch me in #gentoo of OPN if you think forum is not interactive enough. :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum