Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to reduce your download traffic by 75% or more
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4 ... 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Thu Sep 02, 2004 6:39 am    Post subject: Reply with quote

Quote:


about imlib2-1.1.1 to imlib2-1.1.2 - I wonder what you mean, since there is a dtu for that on the server


Perhaps I was one of the first ones to try. When I tried the server reported it couldn't download imlib2-1.1.2...

Quote:

So that's the reason why the server could not find the file (the original URL is not passed to the server - and it would not make sense to do so)


Could you elaborate why it doesn't make sense to pass the original url? How would you then expect ut2004 delta to be generated?

What a bout the no third party mirrors? Can that be changed? Would it make sense?


Right now I am experiencing something even stranger:

(edit) bug fixed so I removed hunk of text.

Why the heck does my request go up in the queue instead of down?


Last edited by PrakashP on Mon Sep 06, 2004 5:40 am; edited 2 times in total
Back to top
View user's profile Send private message
pleusicles
n00b
n00b


Joined: 26 May 2004
Posts: 44

PostPosted: Thu Sep 02, 2004 7:05 am    Post subject: Reply with quote

blackpenguin wrote:

Neither portage nor deltup is wrong in that case. As you can check on bugs.gentoo.org - the font-files on mplayerhq change every few months without any obvious reason - therefore the md5sum often changes.


Thank you for the answer. Yesterday I was chatting with wiebel on freenode about my problem and he helped to clear the situation for me. We agreed that the exclusion list you mentioned as a planned feature would be the solution for such rare problems; the other 99% Just Works :)

Another feature I think would be great is the ability to set a minimum file size, under which the original file is simply downloaded without dtu even if the user has a previous version. Of course, a small file is a small load for the server, but with such a setting the user could set the size where _for him/her_ the possible difference between the size of the original and the dtu begins to outweigh the possible time needed to getting to the front of the queue. Time, I think, is an important matter e. g. for most modem users.

Or, optionally, a setting where you could set a "HIGH QUEUE RATING" (a' la *mule) above which you simply download the original.

These are, of course, just ideas; they may have drawbacks.

Again, thanks for your work!
Back to top
View user's profile Send private message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Thu Sep 02, 2004 11:53 am    Post subject: Reply with quote

Something is really fsked up. I tried again to get memtest86+ 1.25 (or more specifically memtest86+-1.20.tar.gz-memtest86+-1.25.tar.gz.dtu). First I got the message that the server is creating the dtu, then after three tries, I was queued and the postiton rised steadily. What is this?
Back to top
View user's profile Send private message
ssvb
Tux's lil' helper
Tux's lil' helper


Joined: 06 Nov 2003
Posts: 96

PostPosted: Thu Sep 02, 2004 9:22 pm    Post subject: Reply with quote

What about using bdelta for making patches (available from deltup home page)? It is much more efficient than xdelta.
Back to top
View user's profile Send private message
Wiebel
n00b
n00b


Joined: 05 Feb 2003
Posts: 59
Location: Bremen, Germany

PostPosted: Fri Sep 03, 2004 2:57 am    Post subject: Reply with quote

The problem that the queue is going up instead of down is known and blackpenguin is currently working on it. Don't forget it's a beta. Things will improve and this special problem is handled with highest priority, so please be patient, the dynamic-deltup-server is on it's way. :wink:
Back to top
View user's profile Send private message
JimmyW
Tux's lil' helper
Tux's lil' helper


Joined: 28 Sep 2002
Posts: 119
Location: Sweden

PostPosted: Fri Sep 03, 2004 10:58 am    Post subject: Not working for me. Reply with quote

Delta givs a message:

cannot read file /tmp/.zpItqY/000000

with random path every time, im running gcc 2.4.1 on amd64, any sucsess on this anyone?
Back to top
View user's profile Send private message
eNTi
Veteran
Veteran


Joined: 20 Oct 2002
Posts: 1011
Location: Salzburg, Austria

PostPosted: Fri Sep 03, 2004 3:24 pm    Post subject: Reply with quote

this is working great. good job :).
_________________
If you fall off a cliff, you might as well try to fly. After all, you got nothing to lose.

-- John Sheridan - Babylon 5, Season 4
Back to top
View user's profile Send private message
geekmug
n00b
n00b


Joined: 02 Apr 2004
Posts: 39

PostPosted: Sat Sep 04, 2004 6:40 am    Post subject: Reply with quote

There is a problem with "REMOVE_OLD" on some ebuilds. That feature would be really nice 'cause then I don't have to manually figure out what distfiles to remove. But the problem is that some ebuilds depend on previous versions of the tarball to be available and unpacked, where getdelta will fetch the earliest version required, and then recognize that it can be patched against, and thus delete the older versions sequentially towards the latest version. Not really sure what if anything can be done about this.

An example would be >sys-devel/libtool-1.3 since they will install =libtool-1.3 at the same time.. perhaps this is really a problem with portage, since I should think libtool should be a slotted ebuild then. *shrug*
Back to top
View user's profile Send private message
ts77
Apprentice
Apprentice


Joined: 23 Mar 2003
Posts: 200
Location: Berlin, Germany

PostPosted: Sat Sep 04, 2004 8:12 am    Post subject: Reply with quote

its working great thanks a lot :).

could you please make the "retry-time" (time until it tries again to download a queued dtu, which is currently 10 seconds) configurable?
I hacked my script to have it at 30 seconds as I never got the dtu built in 10 seconds really ;)
Back to top
View user's profile Send private message
yahewitt
n00b
n00b


Joined: 20 Oct 2003
Posts: 12

PostPosted: Sun Sep 05, 2004 2:11 pm    Post subject: Reply with quote

Any chance of making the getdelta.sh to work with http-replicator? An initial look seemed to suggest we just need to look in the http-replicator cache for the appropriate .tar.{gz, bz2} file instead of the DISTDIR, since that is periodically cleaned out by 'repcacheman'.

I was thinking the 2 methods together would *really* minimise downloads for those of us with multiple Gentoo boxes on a LAN - only downloading the delta's (and only once!) to a cache, then serving the whole ebuild via replicator across the local network since we have plenty of (unmetered!) bandwidth to spare there. Seem sensible?

:?
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Mon Sep 06, 2004 1:52 am    Post subject: Reply with quote

PrakashKC wrote:

Could you elaborate why it doesn't make sense to pass the original url? How would you then expect ut2004 delta to be generated?

The decision from where to get files from must be done for 2 reasons:
1st: only the server knows which mirrors to prefer - the user would supply a mirror that is near to him/her but the server uses its own preferences.

2nd: the server cannot accept blindly a given URL - probably this would be misused to catch files from arbitrary servers (and arbitrary contents)

PrakashKC wrote:

What a bout the no third party mirrors? Can that be changed? Would it make sense?


the third party mirrors file is maintained by gentoo-devs, not me.
So just putting an additional server to an existing mirror-group would not help as long as the mirrors of that group do not really hold the file.


Last not least, about the increasing queuepos-no.
It was caused by a bug that didn't appear before the public beta but is fixed now.


bp
Back to top
View user's profile Send private message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Mon Sep 06, 2004 5:39 am    Post subject: Reply with quote

Quote:

2nd: the server cannot accept blindly a given URL - probably this would be misused to catch files from arbitrary servers (and arbitrary contents)


What about this: Beside name and a mirror, you transmit the name of ebuild to the server. The server should also keep a portage tree (or at least fetch that ebuild in question from an official portage sync server) so that the server could check for urls in the original ebuild. In this case misusing is highly unprobable and at least for the official portage we could get our deltas.

I have the problem that I use latest ~x86 stuff and as such the files are not always on gentoo mirrors when I emerge them, thus your server won't find the files to create deltas. That's why in my case I am seldomly able to get deltas...
Back to top
View user's profile Send private message
fng
n00b
n00b


Joined: 03 Oct 2003
Posts: 38
Location: Rotselaar, Belgium

PostPosted: Mon Sep 06, 2004 4:13 pm    Post subject: Reply with quote

I dont know where to post bugs with deltup, so i'll just post them here :

Code:
>>> emerge (4 of 20) games-misc/fortune-mod-osfortune-1 to /
>>> Downloading http://www.dibona.com/opensources/osfortune.tar.gz
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

ls: os*: No such file or directory
No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

--18:09:07--  http://www.dibona.com/opensources/osfortune.tar.gz
           => `osfortune.tar.gz'
Resolving www.dibona.com... 66.93.180.23, 209.81.9.22
Connecting to www.dibona.com[66.93.180.23]:80... connected.
HTTP request sent, awaiting response... 404 Not Found
18:09:08 ERROR 404: Not Found.

!!! Couldn't download osfortune.tar.gz. Aborting.

_________________
Play More Promode
Back to top
View user's profile Send private message
geekmug
n00b
n00b


Joined: 02 Apr 2004
Posts: 39

PostPosted: Tue Sep 07, 2004 3:30 am    Post subject: Reply with quote

The problem with osfortune is not getdelta's fault.. that really is a 404 url for the source package.. however it is sitting on the gentoo mirrors.. so you should set that up in /etc/make.conf and it will install fine.

mirrorselect -a -s3 -b10 -o >> /etc/make.conf

Which will take a bit find the best 3 for your host, or you can select them by hand and place them in make.conf yourself.
Back to top
View user's profile Send private message
real_triad
n00b
n00b


Joined: 13 Apr 2004
Posts: 7
Location: Pekanbaru, Riau Indonesia

PostPosted: Tue Sep 07, 2004 4:30 am    Post subject: Reply with quote

agree with u yahewitt,
and btw why not making this dynamic deltup server as new ebuild in portage
_________________
(:> Gentoo where there Best and Freedom combine
Back to top
View user's profile Send private message
Trejkaz
Guru
Guru


Joined: 14 Nov 2002
Posts: 479
Location: Sydney, Australia

PostPosted: Tue Sep 07, 2004 4:44 am    Post subject: Reply with quote

The first mirror on my GENTOO_MIRRORS is a machine inside my local network, so downloading the whole file from the first mirror is actually cheaper than downloading the delta from the internet. But downloading the delta from the internet is still cheaper than downloading the whole file from the second mirror.

I wonder how hard it would be to whack some more logic into this script, the idea of delta downloading is very appealing to me, as long as it doesn't clash with my existing methods of download savings. :-)

Edit: I think the trick would be to keep a list of instructions...
1. Attempt mirror http://server/gentoo
2. Attempt xpatch
3. Attempt mirror http://...
4. Attempt real site.

The best way for this to work would be if some sort of magic URL could be put into the GENTOO_MIRRORS variable, that would be magically picked up by portage and treated as a deltup server. Then anyone can choose how far along it goes. :-)


Last edited by Trejkaz on Tue Sep 07, 2004 2:11 pm; edited 1 time in total
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Tue Sep 07, 2004 2:01 pm    Post subject: Reply with quote

It works fantasticly!!

Also a good idea to add "-c" to your FETCH variable. This will allow delta to resume the delta files.
It doesn't effect portages resume it any way!!

I hope this gets integrated into portage. i just can't wait.

Now i feel really bad about deleting all the old distfiles!!! :( :(

PS: one more feature you could add is to allow number of old files to remain. for example, i want to keep atleast 2 versions of gaim files. If a file is older then two versions it should be deleted.
That would be nice feature. Sometimes you want to downgrade to the previous version after an unsuccessful upgrade. Keeping atleast two versions is nice. (or let the user choose how many old versions to keep)
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
Wiebel
n00b
n00b


Joined: 05 Feb 2003
Posts: 59
Location: Bremen, Germany

PostPosted: Wed Sep 08, 2004 2:32 pm    Post subject: Reply with quote

@Trejkaz:
A completly different way to achieve what you want, would be to export the distfiles-folder via nfs, so you can directly use getdelta without any modifications. Also every other machine in the local net would benefit from the new build distfiles.

@MooktaKiNG:
The upcoming release will have a dont_remove file where you can paste files you want to keep.
You are free to choose whatever FETCH programm you like so simply add the -c parameter if you like.
Back to top
View user's profile Send private message
Regor
Guru
Guru


Joined: 06 Aug 2002
Posts: 545
Location: 39° 2' 48" N, 120° 59' 2" W

PostPosted: Thu Sep 09, 2004 7:59 am    Post subject: Reply with quote

Wow! If you could see me, I'm doing a major happy dance right now! I just updated samba and a 15mb download became 3mb. This is fantastic! I can't wait to try it on an even larger package (OOo comes to mind).

I, a long-suffering Gentoo over dialup user, salute you :D
_________________
Sometimes the appropriate response to reality is to go insane.
-Philip K. Dick, Valis
Back to top
View user's profile Send private message
pqwy
n00b
n00b


Joined: 28 Jul 2004
Posts: 9

PostPosted: Thu Sep 09, 2004 10:03 pm    Post subject: Reply with quote

Yat another dialup user humbly bows before the dark overlords of deltup!

Up until now, I had to use some scripts to fetch to a college machine, then harvest to an IDE harddrive I tugged around... And now, I can comfortably emerge Xorg! There are places where deltup makes for a quantum leap.
Back to top
View user's profile Send private message
Pkunkage
n00b
n00b


Joined: 10 Sep 2004
Posts: 4

PostPosted: Fri Sep 10, 2004 9:49 am    Post subject: A small update to the download script Reply with quote

Firstly i would like to congratulate blackpenguin a very well done project.

Now that the queue problems are once and for all sorted out , i would like to present a fix to a problem lots of users have faced with the current download script.

Suppose the existing file which is to be upgrade is corrupt or incomplete , deltup will still try to download a dtu patch for it , and then fail after going trhrough the download with the "original file is corrupt" message. I recently had download an 8 meg dtu and after downloading it told "original file is corrupt" and proceeded to download the new file.

I have implemented some local checks into the get_delta.sh script which check the digest files if availabe and if not , test the tarball for corruption and then if the file is corrupt , NOT download the dtu , but instead ask the user to delete the old file and retry.

To install this updated script ----
Firstly , please update your /etc/make.conf with

FETCHCOMMAND="/usr/bin/getdelta_pkunk.sh \${URI} \${FILESDIR} \${PF}"

Goto http://bugs.gentoo.org/show_bug.cgi?id=63525
And download the attachment "Updated get_delta.sh script" and save it to /usr/bin and chmod 755 /usr/bin/getdelta_pkunk.sh

Reply to to this post or in bugzilla if you notice any problems.
Back to top
View user's profile Send private message
ep98
Apprentice
Apprentice


Joined: 08 Jul 2003
Posts: 232
Location: Bulgaria

PostPosted: Sat Sep 11, 2004 6:37 am    Post subject: Reply with quote

where I'm wrong
when i type;

emerge search deltup
emerge says applications found 0
_________________
If u cant give advice, dont send the helpless to the HOWTO's way or Google.com way.
Back to top
View user's profile Send private message
eNTi
Veteran
Veteran


Joined: 20 Oct 2002
Posts: 1011
Location: Salzburg, Austria

PostPosted: Sat Sep 11, 2004 10:09 am    Post subject: Reply with quote

ep98 wrote:
where I'm wrong
when i type;

emerge search deltup
emerge says applications found 0

that's because why you need to follow the instructions in the 3rd post.
_________________
If you fall off a cliff, you might as well try to fly. After all, you got nothing to lose.

-- John Sheridan - Babylon 5, Season 4
Back to top
View user's profile Send private message
Regor
Guru
Guru


Joined: 06 Aug 2002
Posts: 545
Location: 39° 2' 48" N, 120° 59' 2" W

PostPosted: Sat Sep 11, 2004 10:26 am    Post subject: Reply with quote

The ebuild isn't in portage (though I get the impression it used to be) you have to download it from blackpenguin's website and install it manually.
_________________
Sometimes the appropriate response to reality is to go insane.
-Philip K. Dick, Valis
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Sep 12, 2004 2:11 am    Post subject: NEW VERSION OF getdelta.sh RELEASED Reply with quote

THANKS for all your feedback and bug reports!

getdelta.sh version 0.5.3 released

As you might have noticed already, a new version of getdelta.sh is available now.
This new version fixes a severe bug with updating files beginning with "lib" in their name - therefore I've forced the update - the server will not accept getdelta.sh 0.5.2.x anymore.

I've tried to implement most of the requested features like
- LOCAL_MIRROR to download from before requesting a dtu
- the possibility to exclude specific files from REMOVE_OLD,
- Pkunk's corruption check of previous version before downloading a dtu,
- a separate config-file (actually no variables to be set in the script anymore (oh, yes, the name of the config-file if you want to change it :))
- a configurable maximum acceptable queue-position
- a configurable retry time for queued requests

Have fun with the new getdelta.sh
and please dont stop to give feedback :)

bp

http://linux01.gwdg.de/~nlissne/getdelta.sh
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3, 4 ... 9, 10, 11  Next
Page 3 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum