How to reduce your download traffic by 75% or more

[PrakashP]

about imlib2-1.1.1 to imlib2-1.1.2 - I wonder what you mean, since there is a dtu for that on the server

Perhaps I was one of the first ones to try. When I tried the server reported it couldn't download imlib2-1.1.2...

So that's the reason why the server could not find the file (the original URL is not passed to the server - and it would not make sense to do so)

Could you elaborate why it doesn't make sense to pass the original url? How would you then expect ut2004 delta to be generated?

What a bout the no third party mirrors? Can that be changed? Would it make sense?


Right now I am experiencing something even stranger:

(edit) bug fixed so I removed hunk of text.

Why the heck does my request go up in the queue instead of down?

[pleusicles]

Neither portage nor deltup is wrong in that case. As you can check on bugs.gentoo.org - the font-files on mplayerhq change every few months without any obvious reason - therefore the md5sum often changes.

Thank you for the answer. Yesterday I was chatting with wiebel on freenode about my problem and he helped to clear the situation for me. We agreed that the exclusion list you mentioned as a planned feature would be the solution for such rare problems; the other 99% Just Works

Another feature I think would be great is the ability to set a minimum file size, under which the original file is simply downloaded without dtu even if the user has a previous version. Of course, a small file is a small load for the server, but with such a setting the user could set the size where _for him/her_ the possible difference between the size of the original and the dtu begins to outweigh the possible time needed to getting to the front of the queue. Time, I think, is an important matter e. g. for most modem users.

Or, optionally, a setting where you could set a "HIGH QUEUE RATING" (a' la *mule) above which you simply download the original.

These are, of course, just ideas; they may have drawbacks.

Again, thanks for your work!

[PrakashP]

Something is really fsked up. I tried again to get memtest86+ 1.25 (or more specifically memtest86+-1.20.tar.gz-memtest86+-1.25.tar.gz.dtu). First I got the message that the server is creating the dtu, then after three tries, I was queued and the postiton rised steadily. What is this?

[ssvb]

What about using bdelta for making patches (available from deltup home page)? It is much more efficient than xdelta.

[Wiebel]

The problem that the queue is going up instead of down is known and blackpenguin is currently working on it. Don't forget it's a beta. Things will improve and this special problem is handled with highest priority, so please be patient, the dynamic-deltup-server is on it's way.

[JimmyW]

Delta givs a message:

cannot read file /tmp/.zpItqY/000000

with random path every time, im running gcc 2.4.1 on amd64, any sucsess on this anyone?

[eNTi]

this is working great. good job .

[geekmug]

There is a problem with "REMOVE_OLD" on some ebuilds. That feature would be really nice 'cause then I don't have to manually figure out what distfiles to remove. But the problem is that some ebuilds depend on previous versions of the tarball to be available and unpacked, where getdelta will fetch the earliest version required, and then recognize that it can be patched against, and thus delete the older versions sequentially towards the latest version. Not really sure what if anything can be done about this.

An example would be >sys-devel/libtool-1.3 since they will install =libtool-1.3 at the same time.. perhaps this is really a problem with portage, since I should think libtool should be a slotted ebuild then. *shrug*

[ts77]

its working great thanks a lot .

could you please make the "retry-time" (time until it tries again to download a queued dtu, which is currently 10 seconds) configurable?
I hacked my script to have it at 30 seconds as I never got the dtu built in 10 seconds really

[yahewitt]

Any chance of making the getdelta.sh to work with http-replicator? An initial look seemed to suggest we just need to look in the http-replicator cache for the appropriate .tar.{gz, bz2} file instead of the DISTDIR, since that is periodically cleaned out by 'repcacheman'.

I was thinking the 2 methods together would *really* minimise downloads for those of us with multiple Gentoo boxes on a LAN - only downloading the delta's (and only once!) to a cache, then serving the whole ebuild via replicator across the local network since we have plenty of (unmetered!) bandwidth to spare there. Seem sensible?

[blackpenguin]

Could you elaborate why it doesn't make sense to pass the original url? How would you then expect ut2004 delta to be generated?

The decision from where to get files from must be done for 2 reasons:
1st: only the server knows which mirrors to prefer - the user would supply a mirror that is near to him/her but the server uses its own preferences.

2nd: the server cannot accept blindly a given URL - probably this would be misused to catch files from arbitrary servers (and arbitrary contents)

What a bout the no third party mirrors? Can that be changed? Would it make sense?

the third party mirrors file is maintained by gentoo-devs, not me.
So just putting an additional server to an existing mirror-group would not help as long as the mirrors of that group do not really hold the file.


Last not least, about the increasing queuepos-no.
It was caused by a bug that didn't appear before the public beta but is fixed now.


bp

[PrakashP]

2nd: the server cannot accept blindly a given URL - probably this would be misused to catch files from arbitrary servers (and arbitrary contents)

What about this: Beside name and a mirror, you transmit the name of ebuild to the server. The server should also keep a portage tree (or at least fetch that ebuild in question from an official portage sync server) so that the server could check for urls in the original ebuild. In this case misusing is highly unprobable and at least for the official portage we could get our deltas.

I have the problem that I use latest ~x86 stuff and as such the files are not always on gentoo mirrors when I emerge them, thus your server won't find the files to create deltas. That's why in my case I am seldomly able to get deltas...

[fng]

I dont know where to post bugs with deltup, so i'll just post them here :

Code: Select all

>>> emerge (4 of 20) games-misc/fortune-mod-osfortune-1 to /
>>> Downloading http://www.dibona.com/opensources/osfortune.tar.gz
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

ls: os*: No such file or directory
No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

--18:09:07--  http://www.dibona.com/opensources/osfortune.tar.gz
           => `osfortune.tar.gz'
Resolving www.dibona.com... 66.93.180.23, 209.81.9.22
Connecting to www.dibona.com[66.93.180.23]:80... connected.
HTTP request sent, awaiting response... 404 Not Found
18:09:08 ERROR 404: Not Found.

!!! Couldn't download osfortune.tar.gz. Aborting.

[geekmug]

The problem with osfortune is not getdelta's fault.. that really is a 404 url for the source package.. however it is sitting on the gentoo mirrors.. so you should set that up in /etc/make.conf and it will install fine.

mirrorselect -a -s3 -b10 -o >> /etc/make.conf

Which will take a bit find the best 3 for your host, or you can select them by hand and place them in make.conf yourself.

[real_triad]

agree with u yahewitt,
and btw why not making this dynamic deltup server as new ebuild in portage

[Trejkaz]

The first mirror on my GENTOO_MIRRORS is a machine inside my local network, so downloading the whole file from the first mirror is actually cheaper than downloading the delta from the internet. But downloading the delta from the internet is still cheaper than downloading the whole file from the second mirror.

I wonder how hard it would be to whack some more logic into this script, the idea of delta downloading is very appealing to me, as long as it doesn't clash with my existing methods of download savings.

Edit: I think the trick would be to keep a list of instructions...
1. Attempt mirror http://server/gentoo
2. Attempt xpatch
3. Attempt mirror http://...
4. Attempt real site.

The best way for this to work would be if some sort of magic URL could be put into the GENTOO_MIRRORS variable, that would be magically picked up by portage and treated as a deltup server. Then anyone can choose how far along it goes.

[MooktaKiNG]

It works fantasticly!!

Also a good idea to add "-c" to your FETCH variable. This will allow delta to resume the delta files.
It doesn't effect portages resume it any way!!

I hope this gets integrated into portage. i just can't wait.

Now i feel really bad about deleting all the old distfiles!!!

PS: one more feature you could add is to allow number of old files to remain. for example, i want to keep atleast 2 versions of gaim files. If a file is older then two versions it should be deleted.
That would be nice feature. Sometimes you want to downgrade to the previous version after an unsuccessful upgrade. Keeping atleast two versions is nice. (or let the user choose how many old versions to keep)

[Wiebel]

@Trejkaz:
A completly different way to achieve what you want, would be to export the distfiles-folder via nfs, so you can directly use getdelta without any modifications. Also every other machine in the local net would benefit from the new build distfiles.

@MooktaKiNG:
The upcoming release will have a dont_remove file where you can paste files you want to keep.
You are free to choose whatever FETCH programm you like so simply add the -c parameter if you like.

[Regor]

Wow! If you could see me, I'm doing a major happy dance right now! I just updated samba and a 15mb download became 3mb. This is fantastic! I can't wait to try it on an even larger package (OOo comes to mind).

I, a long-suffering Gentoo over dialup user, salute you

[pqwy]

Yat another dialup user humbly bows before the dark overlords of deltup!

Up until now, I had to use some scripts to fetch to a college machine, then harvest to an IDE harddrive I tugged around... And now, I can comfortably emerge Xorg! There are places where deltup makes for a quantum leap.

[Pkunkage]

Firstly i would like to congratulate blackpenguin a very well done project.

Now that the queue problems are once and for all sorted out , i would like to present a fix to a problem lots of users have faced with the current download script.

Suppose the existing file which is to be upgrade is corrupt or incomplete , deltup will still try to download a dtu patch for it , and then fail after going trhrough the download with the "original file is corrupt" message. I recently had download an 8 meg dtu and after downloading it told "original file is corrupt" and proceeded to download the new file.

I have implemented some local checks into the get_delta.sh script which check the digest files if availabe and if not , test the tarball for corruption and then if the file is corrupt , NOT download the dtu , but instead ask the user to delete the old file and retry.

To install this updated script ----
Firstly , please update your /etc/make.conf with

FETCHCOMMAND="/usr/bin/getdelta_pkunk.sh \${URI} \${FILESDIR} \${PF}"

Goto http://bugs.gentoo.org/show_bug.cgi?id=63525
And download the attachment "Updated get_delta.sh script" and save it to /usr/bin and chmod 755 /usr/bin/getdelta_pkunk.sh

Reply to to this post or in bugzilla if you notice any problems.

[ep98]

where I'm wrong
when i type;

emerge search deltup
emerge says applications found 0

[eNTi]

where I'm wrong
when i type;

emerge search deltup
emerge says applications found 0

that's because why you need to follow the instructions in the 3rd post.

[Regor]

The ebuild isn't in portage (though I get the impression it used to be) you have to download it from blackpenguin's website and install it manually.

[blackpenguin]

THANKS for all your feedback and bug reports!

getdelta.sh version 0.5.3 released

As you might have noticed already, a new version of getdelta.sh is available now.
This new version fixes a severe bug with updating files beginning with "lib" in their name - therefore I've forced the update - the server will not accept getdelta.sh 0.5.2.x anymore.

I've tried to implement most of the requested features like
- LOCAL_MIRROR to download from before requesting a dtu
- the possibility to exclude specific files from REMOVE_OLD,
- Pkunk's corruption check of previous version before downloading a dtu,
- a separate config-file (actually no variables to be set in the script anymore (oh, yes, the name of the config-file if you want to change it )
- a configurable maximum acceptable queue-position
- a configurable retry time for queued requests

Have fun with the new getdelta.sh
and please dont stop to give feedback

bp

http://linux01.gwdg.de/~nlissne/getdelta.sh