View previous topic :: View next topic |
Author |
Message |
PrakashP Veteran
Joined: 27 Oct 2003 Posts: 1249 Location: C.C.A.A., Germania
|
Posted: Thu Sep 02, 2004 6:39 am Post subject: |
|
|
Quote: |
about imlib2-1.1.1 to imlib2-1.1.2 - I wonder what you mean, since there is a dtu for that on the server
|
Perhaps I was one of the first ones to try. When I tried the server reported it couldn't download imlib2-1.1.2...
Quote: |
So that's the reason why the server could not find the file (the original URL is not passed to the server - and it would not make sense to do so)
|
Could you elaborate why it doesn't make sense to pass the original url? How would you then expect ut2004 delta to be generated?
What a bout the no third party mirrors? Can that be changed? Would it make sense?
Right now I am experiencing something even stranger:
(edit) bug fixed so I removed hunk of text.
Why the heck does my request go up in the queue instead of down?
Last edited by PrakashP on Mon Sep 06, 2004 5:40 am; edited 2 times in total |
|
Back to top |
|
|
pleusicles n00b
Joined: 26 May 2004 Posts: 44
|
Posted: Thu Sep 02, 2004 7:05 am Post subject: |
|
|
blackpenguin wrote: |
Neither portage nor deltup is wrong in that case. As you can check on bugs.gentoo.org - the font-files on mplayerhq change every few months without any obvious reason - therefore the md5sum often changes.
|
Thank you for the answer. Yesterday I was chatting with wiebel on freenode about my problem and he helped to clear the situation for me. We agreed that the exclusion list you mentioned as a planned feature would be the solution for such rare problems; the other 99% Just Works
Another feature I think would be great is the ability to set a minimum file size, under which the original file is simply downloaded without dtu even if the user has a previous version. Of course, a small file is a small load for the server, but with such a setting the user could set the size where _for him/her_ the possible difference between the size of the original and the dtu begins to outweigh the possible time needed to getting to the front of the queue. Time, I think, is an important matter e. g. for most modem users.
Or, optionally, a setting where you could set a "HIGH QUEUE RATING" (a' la *mule) above which you simply download the original.
These are, of course, just ideas; they may have drawbacks.
Again, thanks for your work! |
|
Back to top |
|
|
PrakashP Veteran
Joined: 27 Oct 2003 Posts: 1249 Location: C.C.A.A., Germania
|
Posted: Thu Sep 02, 2004 11:53 am Post subject: |
|
|
Something is really fsked up. I tried again to get memtest86+ 1.25 (or more specifically memtest86+-1.20.tar.gz-memtest86+-1.25.tar.gz.dtu). First I got the message that the server is creating the dtu, then after three tries, I was queued and the postiton rised steadily. What is this? |
|
Back to top |
|
|
ssvb Tux's lil' helper
Joined: 06 Nov 2003 Posts: 96
|
Posted: Thu Sep 02, 2004 9:22 pm Post subject: |
|
|
What about using bdelta for making patches (available from deltup home page)? It is much more efficient than xdelta. |
|
Back to top |
|
|
Wiebel n00b
Joined: 05 Feb 2003 Posts: 60 Location: Bad Homburg, Germany
|
Posted: Fri Sep 03, 2004 2:57 am Post subject: |
|
|
The problem that the queue is going up instead of down is known and blackpenguin is currently working on it. Don't forget it's a beta. Things will improve and this special problem is handled with highest priority, so please be patient, the dynamic-deltup-server is on it's way. |
|
Back to top |
|
|
JimmyW Tux's lil' helper
Joined: 28 Sep 2002 Posts: 119 Location: Sweden
|
Posted: Fri Sep 03, 2004 10:58 am Post subject: Not working for me. |
|
|
Delta givs a message:
cannot read file /tmp/.zpItqY/000000
with random path every time, im running gcc 2.4.1 on amd64, any sucsess on this anyone? |
|
Back to top |
|
|
eNTi Veteran
Joined: 20 Oct 2002 Posts: 1011 Location: Salzburg, Austria
|
Posted: Fri Sep 03, 2004 3:24 pm Post subject: |
|
|
this is working great. good job . _________________ If you fall off a cliff, you might as well try to fly. After all, you got nothing to lose.
-- John Sheridan - Babylon 5, Season 4 |
|
Back to top |
|
|
geekmug n00b
Joined: 02 Apr 2004 Posts: 39
|
Posted: Sat Sep 04, 2004 6:40 am Post subject: |
|
|
There is a problem with "REMOVE_OLD" on some ebuilds. That feature would be really nice 'cause then I don't have to manually figure out what distfiles to remove. But the problem is that some ebuilds depend on previous versions of the tarball to be available and unpacked, where getdelta will fetch the earliest version required, and then recognize that it can be patched against, and thus delete the older versions sequentially towards the latest version. Not really sure what if anything can be done about this.
An example would be >sys-devel/libtool-1.3 since they will install =libtool-1.3 at the same time.. perhaps this is really a problem with portage, since I should think libtool should be a slotted ebuild then. *shrug* |
|
Back to top |
|
|
ts77 Apprentice
Joined: 23 Mar 2003 Posts: 200 Location: Berlin, Germany
|
Posted: Sat Sep 04, 2004 8:12 am Post subject: |
|
|
its working great thanks a lot .
could you please make the "retry-time" (time until it tries again to download a queued dtu, which is currently 10 seconds) configurable?
I hacked my script to have it at 30 seconds as I never got the dtu built in 10 seconds really |
|
Back to top |
|
|
yahewitt n00b
Joined: 20 Oct 2003 Posts: 12
|
Posted: Sun Sep 05, 2004 2:11 pm Post subject: |
|
|
Any chance of making the getdelta.sh to work with http-replicator? An initial look seemed to suggest we just need to look in the http-replicator cache for the appropriate .tar.{gz, bz2} file instead of the DISTDIR, since that is periodically cleaned out by 'repcacheman'.
I was thinking the 2 methods together would *really* minimise downloads for those of us with multiple Gentoo boxes on a LAN - only downloading the delta's (and only once!) to a cache, then serving the whole ebuild via replicator across the local network since we have plenty of (unmetered!) bandwidth to spare there. Seem sensible?
|
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Mon Sep 06, 2004 1:52 am Post subject: |
|
|
PrakashKC wrote: |
Could you elaborate why it doesn't make sense to pass the original url? How would you then expect ut2004 delta to be generated?
|
The decision from where to get files from must be done for 2 reasons:
1st: only the server knows which mirrors to prefer - the user would supply a mirror that is near to him/her but the server uses its own preferences.
2nd: the server cannot accept blindly a given URL - probably this would be misused to catch files from arbitrary servers (and arbitrary contents)
PrakashKC wrote: |
What a bout the no third party mirrors? Can that be changed? Would it make sense?
|
the third party mirrors file is maintained by gentoo-devs, not me.
So just putting an additional server to an existing mirror-group would not help as long as the mirrors of that group do not really hold the file.
Last not least, about the increasing queuepos-no.
It was caused by a bug that didn't appear before the public beta but is fixed now.
bp |
|
Back to top |
|
|
PrakashP Veteran
Joined: 27 Oct 2003 Posts: 1249 Location: C.C.A.A., Germania
|
Posted: Mon Sep 06, 2004 5:39 am Post subject: |
|
|
Quote: |
2nd: the server cannot accept blindly a given URL - probably this would be misused to catch files from arbitrary servers (and arbitrary contents)
|
What about this: Beside name and a mirror, you transmit the name of ebuild to the server. The server should also keep a portage tree (or at least fetch that ebuild in question from an official portage sync server) so that the server could check for urls in the original ebuild. In this case misusing is highly unprobable and at least for the official portage we could get our deltas.
I have the problem that I use latest ~x86 stuff and as such the files are not always on gentoo mirrors when I emerge them, thus your server won't find the files to create deltas. That's why in my case I am seldomly able to get deltas... |
|
Back to top |
|
|
fng n00b
Joined: 03 Oct 2003 Posts: 38 Location: Rotselaar, Belgium
|
Posted: Mon Sep 06, 2004 4:13 pm Post subject: |
|
|
I dont know where to post bugs with deltup, so i'll just post them here :
Code: | >>> emerge (4 of 20) games-misc/fortune-mod-osfortune-1 to /
>>> Downloading http://www.dibona.com/opensources/osfortune.tar.gz
Checking if a Thirdparty-Mirror is used ...
No third-party mirror.
Searching for a previosly downloaded file in /usr/portage/distfiles
ls: os*: No such file or directory
No old version of the requested file found.
The dtu could not be fetched, downloading full file from original URL
--18:09:07-- http://www.dibona.com/opensources/osfortune.tar.gz
=> `osfortune.tar.gz'
Resolving www.dibona.com... 66.93.180.23, 209.81.9.22
Connecting to www.dibona.com[66.93.180.23]:80... connected.
HTTP request sent, awaiting response... 404 Not Found
18:09:08 ERROR 404: Not Found.
!!! Couldn't download osfortune.tar.gz. Aborting. |
_________________ Play More Promode |
|
Back to top |
|
|
geekmug n00b
Joined: 02 Apr 2004 Posts: 39
|
Posted: Tue Sep 07, 2004 3:30 am Post subject: |
|
|
The problem with osfortune is not getdelta's fault.. that really is a 404 url for the source package.. however it is sitting on the gentoo mirrors.. so you should set that up in /etc/make.conf and it will install fine.
mirrorselect -a -s3 -b10 -o >> /etc/make.conf
Which will take a bit find the best 3 for your host, or you can select them by hand and place them in make.conf yourself. |
|
Back to top |
|
|
real_triad n00b
Joined: 13 Apr 2004 Posts: 7 Location: Pekanbaru, Riau Indonesia
|
Posted: Tue Sep 07, 2004 4:30 am Post subject: |
|
|
agree with u yahewitt,
and btw why not making this dynamic deltup server as new ebuild in portage _________________ (:> Gentoo where there Best and Freedom combine |
|
Back to top |
|
|
Trejkaz Guru
Joined: 14 Nov 2002 Posts: 479 Location: Sydney, Australia
|
Posted: Tue Sep 07, 2004 4:44 am Post subject: |
|
|
The first mirror on my GENTOO_MIRRORS is a machine inside my local network, so downloading the whole file from the first mirror is actually cheaper than downloading the delta from the internet. But downloading the delta from the internet is still cheaper than downloading the whole file from the second mirror.
I wonder how hard it would be to whack some more logic into this script, the idea of delta downloading is very appealing to me, as long as it doesn't clash with my existing methods of download savings.
Edit: I think the trick would be to keep a list of instructions...
1. Attempt mirror http://server/gentoo
2. Attempt xpatch
3. Attempt mirror http://...
4. Attempt real site.
The best way for this to work would be if some sort of magic URL could be put into the GENTOO_MIRRORS variable, that would be magically picked up by portage and treated as a deltup server. Then anyone can choose how far along it goes.
Last edited by Trejkaz on Tue Sep 07, 2004 2:11 pm; edited 1 time in total |
|
Back to top |
|
|
MooktaKiNG Guru
Joined: 11 Nov 2002 Posts: 326 Location: London, UK
|
Posted: Tue Sep 07, 2004 2:01 pm Post subject: |
|
|
It works fantasticly!!
Also a good idea to add "-c" to your FETCH variable. This will allow delta to resume the delta files.
It doesn't effect portages resume it any way!!
I hope this gets integrated into portage. i just can't wait.
Now i feel really bad about deleting all the old distfiles!!!
PS: one more feature you could add is to allow number of old files to remain. for example, i want to keep atleast 2 versions of gaim files. If a file is older then two versions it should be deleted.
That would be nice feature. Sometimes you want to downgrade to the previous version after an unsuccessful upgrade. Keeping atleast two versions is nice. (or let the user choose how many old versions to keep) _________________ http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router |
|
Back to top |
|
|
Wiebel n00b
Joined: 05 Feb 2003 Posts: 60 Location: Bad Homburg, Germany
|
Posted: Wed Sep 08, 2004 2:32 pm Post subject: |
|
|
@Trejkaz:
A completly different way to achieve what you want, would be to export the distfiles-folder via nfs, so you can directly use getdelta without any modifications. Also every other machine in the local net would benefit from the new build distfiles.
@MooktaKiNG:
The upcoming release will have a dont_remove file where you can paste files you want to keep.
You are free to choose whatever FETCH programm you like so simply add the -c parameter if you like. |
|
Back to top |
|
|
Regor Guru
Joined: 06 Aug 2002 Posts: 545 Location: 39° 2' 48" N, 120° 59' 2" W
|
Posted: Thu Sep 09, 2004 7:59 am Post subject: |
|
|
Wow! If you could see me, I'm doing a major happy dance right now! I just updated samba and a 15mb download became 3mb. This is fantastic! I can't wait to try it on an even larger package (OOo comes to mind).
I, a long-suffering Gentoo over dialup user, salute you _________________ Sometimes the appropriate response to reality is to go insane.
-Philip K. Dick, Valis |
|
Back to top |
|
|
pqwy n00b
Joined: 28 Jul 2004 Posts: 9
|
Posted: Thu Sep 09, 2004 10:03 pm Post subject: |
|
|
Yat another dialup user humbly bows before the dark overlords of deltup!
Up until now, I had to use some scripts to fetch to a college machine, then harvest to an IDE harddrive I tugged around... And now, I can comfortably emerge Xorg! There are places where deltup makes for a quantum leap. |
|
Back to top |
|
|
Pkunkage n00b
Joined: 10 Sep 2004 Posts: 4
|
Posted: Fri Sep 10, 2004 9:49 am Post subject: A small update to the download script |
|
|
Firstly i would like to congratulate blackpenguin a very well done project.
Now that the queue problems are once and for all sorted out , i would like to present a fix to a problem lots of users have faced with the current download script.
Suppose the existing file which is to be upgrade is corrupt or incomplete , deltup will still try to download a dtu patch for it , and then fail after going trhrough the download with the "original file is corrupt" message. I recently had download an 8 meg dtu and after downloading it told "original file is corrupt" and proceeded to download the new file.
I have implemented some local checks into the get_delta.sh script which check the digest files if availabe and if not , test the tarball for corruption and then if the file is corrupt , NOT download the dtu , but instead ask the user to delete the old file and retry.
To install this updated script ----
Firstly , please update your /etc/make.conf with
FETCHCOMMAND="/usr/bin/getdelta_pkunk.sh \${URI} \${FILESDIR} \${PF}"
Goto https://bugs.gentoo.org/show_bug.cgi?id=63525
And download the attachment "Updated get_delta.sh script" and save it to /usr/bin and chmod 755 /usr/bin/getdelta_pkunk.sh
Reply to to this post or in bugzilla if you notice any problems. |
|
Back to top |
|
|
ep98 Apprentice
Joined: 08 Jul 2003 Posts: 232 Location: Bulgaria
|
Posted: Sat Sep 11, 2004 6:37 am Post subject: |
|
|
where I'm wrong
when i type;
emerge search deltup
emerge says applications found 0 _________________ If u cant give advice, dont send the helpless to the HOWTO's way or Google.com way. |
|
Back to top |
|
|
eNTi Veteran
Joined: 20 Oct 2002 Posts: 1011 Location: Salzburg, Austria
|
Posted: Sat Sep 11, 2004 10:09 am Post subject: |
|
|
ep98 wrote: | where I'm wrong
when i type;
emerge search deltup
emerge says applications found 0 |
that's because why you need to follow the instructions in the 3rd post. _________________ If you fall off a cliff, you might as well try to fly. After all, you got nothing to lose.
-- John Sheridan - Babylon 5, Season 4 |
|
Back to top |
|
|
Regor Guru
Joined: 06 Aug 2002 Posts: 545 Location: 39° 2' 48" N, 120° 59' 2" W
|
Posted: Sat Sep 11, 2004 10:26 am Post subject: |
|
|
The ebuild isn't in portage (though I get the impression it used to be) you have to download it from blackpenguin's website and install it manually. _________________ Sometimes the appropriate response to reality is to go insane.
-Philip K. Dick, Valis |
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Sun Sep 12, 2004 2:11 am Post subject: NEW VERSION OF getdelta.sh RELEASED |
|
|
THANKS for all your feedback and bug reports!
getdelta.sh version 0.5.3 released
As you might have noticed already, a new version of getdelta.sh is available now.
This new version fixes a severe bug with updating files beginning with "lib" in their name - therefore I've forced the update - the server will not accept getdelta.sh 0.5.2.x anymore.
I've tried to implement most of the requested features like
- LOCAL_MIRROR to download from before requesting a dtu
- the possibility to exclude specific files from REMOVE_OLD,
- Pkunk's corruption check of previous version before downloading a dtu,
- a separate config-file (actually no variables to be set in the script anymore (oh, yes, the name of the config-file if you want to change it )
- a configurable maximum acceptable queue-position
- a configurable retry time for queued requests
Have fun with the new getdelta.sh
and please dont stop to give feedback
bp
http://linux01.gwdg.de/~nlissne/getdelta.sh |
|
Back to top |
|
|
|