How to reduce your download traffic by 75% or more

[localghost]

SPY_jmr1,
There probably is an incompatability with AMD 64 systems. I don't have the hardware to be able to test it. However, it might very well be xdelta that is at fault here. This can be tested by making a test patch that uses bdelta instead of xdelta. Could you work with me to resolve this problem? Maybe we can meet on IRC.
Thanks

i'm still having my same issue as above... I've remerged xdelta many times, but nohing seems to effect it... I am on an AMD64 arch, if that helps anyone think of a possible solution...

Thanks for anything you can think of (of have thought of )

I'd be glad to work with anyone on patches to this contraption, sure... If you could send me some details in a private message, or on the thread, i'll see what I can do... Sorry to get back to this after such a delay, but being a full time student does that unfortunatly...

I also have an amd64 to tinker with. Can I help?

[slycordinator]

Hey, just emerged this beauty and WOW!. Works like a charm. And me being on dialup will be able to have a more up-to-date system. I emerged a newer version of gettext and saved myself 87% of downloading! . Great work guys!

- bob_111

That's nothing.

I'm updating firefox. I saved 30 MB.
Talk about

[ikelos]

It might help to solve some of these problems, and spread the load of the main server, if the server php source code were available (in whatever state it's currently in), so that other people could work on it.

Is there any chance the author could release this code, or perhaps someone else with time and motivation to reverse engineer it and work up a similar script? I have already emailed the author about it, but he seems very busy and hasn't managed to get a reply out to me. Is there much interest in getting the server source code, and setting up more mirrors? Perhaps it has already been released, if so does someone know where I could find a copy?

Thanks,

Mike 5:)

[leighgiles]

G'day,

I'm trying to use this and I get the 'Error applying patch message'

I've hacked getdelta so it doesn't cleanup the dtu it downloads ( lets call it d.dtu)

then I've created an equivalent patch file using deltup ( lets call it c.dtu)

I used genpatches-2.6-10.08-base.tar.bz2 and genpatches-2.6-11.04-base.tar.bz2

Code: Select all

deltup -m -b 9 genpatches-2.6-10.08-base.tar.bz2 genpatches-2.6-11.04-base.tar.bz2 c.dtu

I bunzipped both dtu's and used cmp to compare them ( bunzip2 creates .out files)

Code: Select all

cmp -b d.dtu.out c.dtu.out
d.dtu.out c.out differ: byte 29438, line 836 is   0 ^@  60 0

I used vi to change the ^@ in d.dtu.out to a 0 and the I did

Code: Select all

deltup -p d.dtu.out

and I got a version of genpatches-2.6-11.04-base.tar.bz2 that was exactly the same size as the one in distfiles
i.e. the edited d.dtu.out worked exactly the same as the patch file I created locally


HTH

Leigh

[haylocki]

Hi, I have to use web-rsync to download portage updates on my pc and when using delta update I get the following output

bash-2.05b# emerge-webrsync
Fetching most recent snapshot
Attempting to fetch file dated: 20050330
Searching for a previously downloaded file in /usr/portage/distfiles

No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

--13:40:06-- http://gentoo.osuosl.org/snapshots/port ... .bz2.md5su
=> `portage-20050330.tar.bz2.md5sum'
Resolving access2... 193.113.72.106
Connecting to access2[193.113.72.106]:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
13:40:06 ERROR 404: Not Found.

Searching for a previously downloaded file in /usr/portage/distfiles

No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

--13:40:07-- http://gentoo.blueyonder.co.uk/snapshot ... 30.tar.bz2
md5sum
=> `portage-20050330.tar.bz2.md5sum'
Resolving access2... 193.113.72.106
Connecting to access2[193.113.72.106]:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
13:40:07 ERROR 404: Not Found.

--- No md5sum present on the mirror. (Not yet available.)
Attempting to fetch file dated: 20050329
Searching for a previously downloaded file in /usr/portage/distfiles
We have following candidates to choose from
portage-20050313.tar.bz2
portage-20050314.tar.bz2
portage-20050315.tar.bz2
portage-20050316.tar.bz2
portage-20050317.tar.bz2
portage-20050318.tar.bz2
portage-20050319.tar.bz2
portage-20050320.tar.bz2
portage-20050321.tar.bz2
portage-20050322.tar.bz2
portage-20050323.tar.bz2
portage-20050324.tar.bz2
portage-20050325.tar.bz2
portage-20050326.tar.bz2
portage-20050327.tar.bz2
portage-20050328.tar.bz2
portage-20050329.tar.bz2

The best of all is ... portage-20050329.tar.bz2

Checking if this file is OK.

grep: /digest-*: No such file or directory
Could not find a digest-file for portage-20050329.tar.bz2. Testing file integriy with tar.

Is it possible to get delta update to ignore web-rsync downloads ?

Cheers Ian

[ferringb]

haylock, why not use emerge-delta-webrsync (blog entry with stats here, thread with related discussion here)...

Snapshots are currently generated off of my box and pushed to gentooexperimental.org, sometime in the next few days I'l be switching over to gentoo infrstructure hardware, with it going live/official in the near future...

[ozmighty]

Is anyone else having problems with created files not passing MD5 checks?

Started happening today for me.

[ferringb]

Is anyone else having problems with created files not passing MD5 checks?

Started happening today for me.

Downgrade to bzip2-1.0.2 (assuming you're running ~arch), and read the compressed md5sum section of glep 25.
It's a known issue with relying on md5'ing the compressed result, which deltup tries sidestepping via having multiple bzip2 binaries around...
On the off chance you're using bzip2 v1.0.2*, what distfile? It was likely compressed with either 0.99*, or 1.0.3* in that case...

Or, something else is occuring. Either way the md5 issue is again rearing it's head with 1.0.2 and 1.0.3

[haylocki]

Hi Ferringb,

I have just tried the emerge-delta-webrsync and it does work ok.

but surely if delta-update affects the running of programs that have already been accepted into the portage tree, it will never be accepted itself, and as this program is so useful it would be a shame.

I thought maybe a patch to check on the files location might solve the problem.

i.e. check to see if the files path is /usr/portage/distfiles and if it's not, just do a normal download.

Cheers Ian

[ferringb]

but surely if delta-update affects the running of programs that have already been accepted into the portage tree, it will never be accepted itself, and as this program is so useful it would be a shame.
I thought maybe a patch to check on the files location might solve the problem.
i.e. check to see if the files path is /usr/portage/distfiles and if it's not, just do a normal download.

It's a matter of dumping the files to a different directory.

Easy enough to modify. Aside from that, if you're still hitting issues with emerge-delta-webrsync, kindly pull the most current version from dev.gentoo.org/~ferringb, and try again.
Not aware of any outstanding bugs at this point

[ryszardzonk]

Bigest saves so far are Firefox-source and win32codecs where in both cases it saved something like 97-99% of download!! I would like to thank everyone involved in getting this script done and available in public for all of us to use. Great work


I have read about the issues with bzip2 v1.03, so I downgraded to the 1.02 version at as stated problems with bzipped files do not occur anymore. I have the issues however with passing the MD5 chceks with gzipeed files like Wine. I was waiting for another succesful and quick download, but I ended up downloading whole 11MB archive instead. Is this a known issue or just the accident at work.

My tar version 1.15.1

Code: Select all

[ebuild   R   ] app-arch/tar-1.15.1  -build -debug +nls -static 0 kB

My gzip version 1.3.5

Code: Select all

[ebuild   R   ] app-arch/gzip-1.3.5-r5  -build -debug +nls -pic -static 0 kB

[Cintra]

Hei ferringb

It had been a number of days since I ran emerge-delta-webrsync, all went well, but my 3.7GB portage partition is now 85% full, so I'm looking at cleaning up in distfiles - the snapshots don't take much space, but there are a number of 18.2 MB portage-2005xxxx.tar.bz2 files, going back to portage-20050404.tar.bz2...

Question is whether emerge-delta-webrsync needs more than the latest one?

Mvh

[ferringb]

Question is whether emerge-delta-webrsync needs more than the latest one?
Mvh

Just the most recent. How's about ya split a patch to have it clean up after itself?

It's needed, just haven't taken the time to do it yet

[blackpenguin]

The problems with bzip2 and wrong filesizes/md5sum is solved

Just update to deltup-0.4.2-r1 -- this ebuild depends on bzip2-1.0.3 and will additonally install a statically linked bzip2-1.0.2 as /usr/bin/bzip_old
The dtu-files of the deltup-servers (yes, servers, we've build a network of 4 servers - so you should not wait in the queue anymore) contain a flag that signals to clients deltup which bzip2-version to use.

For those who are interested in details:

[/b]bzip2-1.0.2 used huffman codes with a maximum length of 20 bits when compressing files, but bzip2-1.0.3 uses a maximum length of 17bit huffman codes.
The format did not change - each version can decompress the files the other version compressed, but obviously with these different rules for compression the resulting files *might* differ between these versions. Actually with each file where bzip2-1.0.2 used even a single huffman-code longer than 17 bits the result is different as compressing the same file with bzip2-1.0.3 - But even bzip2-1.0.2 did not use these long huffman-codes in any file - and if all huffman-codes stay <=17bits both versions compress to the same result. That's the reason why the problem occured with some files only, but not all. Anyway, there has been a similar situation between bzip2-0.9 and bzip2-1.0.0 - and therefore deltup has been prepared for this situation. We did not really need to change any code in deltup or bzip2 - we just use the same scheme that deltup already used with the 0.9 <>1.0 versions - a newer version installed as bzip2 and the old version installed as bzip2_old and a flag inside the dtu-file that tells the clients deltup which version to use. Easy - and working.

Our apologies for all the inconvenience.

As mentioned above....
we have built a network of 4 deltup-servers now - this speeds up the creation of new dtu-files
There's no change needed to use the new servers - the new servers will just transparently start working for you when linux01.gwdg.de is busy. Waiting in the queue with high queue-positions is a thing of the past now. Have fun and enjoy.
best wishes,

blackpenguin

P.S.: oh btw, deltup-0.4.2-r1 has just been submitted by genstef to the portage tree - give it an hour or something to appear in emerge sync

[leighgiles]

Will this work with amd64?

[blackpenguin]

Will this work with amd64?

yes and no. Actually it is not trivial to use it on amd64 - the applications build well, but the dtu-files built on a 32bit-machine cannot be used on a 64bit machine and vice versa.

But fortunately A user reported to Build it successfully with -m32 and a little change in the building process.

Have a look at http://giuly.de/xdelta/ to find an ebuild and feel free to contact Giuly on Freenode's IRC in channel #dynamic-deltup-server if you have any questions about or problems with this inofficial ebuild.

good luck!

[Andersson]

Having deltup depend on the newer bzip2 version, which is masked, makes "emerge -u world" exit with a message about "all packages that could satisfy bzip2 has been masked". I hadn't noticed any problem with bzip, so this was a bit confusing to me, trying to figure out why bzip2 was masked all of a sudden.

Isn't there a more elegant solution to the problem? At least let the user know what is causing the problem. Deltup is one of those programs you install and forget about, and most people don't bother much with masked packages.

[slycordinator]

I think there's a major flaw in the server design...

I am trying to download the updated gettext. It couldn't find any dtu files from any of the servers they've got housing those. So then it'll try to download the full file instead from a different server.

Before when it did this it would automatically start using my "GENTOO_MIRRORS" from make.conf. Now, it sat there for a good 5 minutes trying to download from a server in Russia (I'm in Seattle). All for a 7,550 byte file.

Why does it do this? Doesn't it make more sense to have it go to your "GENTOO_MIRRORS" in this case?

edit:
In fact, upon closer inspection I noticed that in some cases it never tried to download using my GENTOO_MIRRORS at all.

[opensas]

Hi

I've been trying deltup

Go the following problem trying to emerge firefox

Code: Select all

The Server creates the dtu-file NOW
--05:57:02--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=firefox-1.0-source.tar.bz2&want=firefox-1.0.3-source.tar.bz2&url=http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/1.0.3/source/firefox-1.0.3-source.tar.bz2&version=0.6
           => `deltup.php?have=firefox-1.0-source.tar.bz2&want=firefox-1.0.3-source.tar.bz2&url=http:%2F%2Fftp.mozilla.org%2Fpub%2Fmozilla.org%2Ffirefox%2Freleases%2F1.0.3%2Fsource%2Ffirefox-1.0.3-source.tar.bz2&version=0.6'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltup-queued [following]
--05:57:31--  http://134.76.13.21/%7Enlissne/deltup-queued
           => `deltup-queued'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36 [text/plain]

100%[======================================================>] 36            --.--K/s

05:57:41 (351.56 KB/s) - `deltup-queued' saved [36/36]

destination file: firefox-1.0.3-source.tar.bz2

The Server creates the dtu-file NOW

TIMEOUT exceeded.

The dtu could not be fetched, downloading full file from original URL

--05:57:41--  http://distfiles.gentoo.org/distfiles/firefox-1.0.3-source.tar.bz2
           => `firefox-1.0.3-source.tar.bz2'
Resolving distfiles.gentoo.org... 216.165.129.135, 156.56.247.195, 140.211.166.134
Connecting to distfiles.gentoo.org[216.165.129.135]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32,784,300 [application/x-tar]

 2% [>                                                      ] 737,036        1.83K/s  ETA 5:21:26

Exiting on signal 2

these is what I have in my $DISTDIR folder

Code: Select all

bash-2.05b# ls -lhs /mnt/xp/gentoo/distfiles/fire*
 32M -rwxr--r--  1 root root  32M Nov  9 03:52 /mnt/xp/gentoo/distfiles/firefox-1.0-source.tar.bz2
720K -rwxr--r--  1 root root 720K Apr 29 06:05 /mnt/xp/gentoo/distfiles/firefox-1.0.3-source.tar.bz2
bash-2.05b#      

The second file is the partially downloaded one, I tried several times erasing it and then running emerge again

here is my make.conf file

Code: Select all

# DELTUP
FETCHCOMMAND="/usr/bin/getdelta.sh \${URI}"

and my modified getdelta.rc

Code: Select all

# snarf
FETCH="/usr/bin/snarf -b "

Any idea about what's going on???

Shall I try whith another server? And how should I do that?


Saludos and thank you all for your efforts

Sas

PS: I think that taking into account the huge amount of redundancy we get in every release of every app, the deltup approach is a matter of common sense. Do you have any idea how is the integration of this marvellous app with gentoo going? I mean, when are ew going to have an unmasked version of it and an official ebuild of the deltup script???

I would even dare say this thing should be integrated with portage, I mean we should have trees of dtu files along with the full sources.

[blackpenguin]

Having deltup depend on the newer bzip2 version, which is masked, makes "emerge -u world" exit with a message about "all packages that could satisfy bzip2 has been masked". I hadn't noticed any problem with bzip, so this was a bit confusing to me, trying to figure out why bzip2 was masked all of a sudden.

IMHO there is nothing wrong in depending on a ~x86 package for a package that is ~x86 itself. However, there is no other working solution for this at the moment.

Isn't there a more elegant solution to the problem? At least let the user know what is causing the problem. Deltup is one of those programs you install and forget about, and most people don't bother much with masked packages.

Have you actually read my posting above? It says exactly what is causing the problem - feel free to ask questions if something is still not clear.

According to the changelog of bzip2-1.0.3 updating is a good idea anyway. To quote the changelog:
"Further robustification against corrupted compressed data. There are currently no known bitstreams which can cause the decompressor to crash, loop or access memory which does not belong to it. If you are using bzip2 or the library to decompress bitstreams from untrusted sources, an upgrade to 1.0.3 is recommended."

However, changing the state of the ebuild from masked to stable is beyond my control.

I also thought the problems with deltup and bzip2 were already known in this thread... To repeat: The rules for compression changed in bzip2-1.0.3 - therefore when using deltup, some files had wrong sizes and md5sum. Since there is no way to recognize which version have been used to compress a file the dynamic-deltup-servers now tag files that have to be compressed using the old version of bzip. Therefore you have to have two bzip2-versions - "bzip2" and "bzip2_old" -- If you really do not want to update to bzip2-1.0.3 - for whatever reason - than make a link bzip2_old that points to your bzip2 executable. But do not whine about wrong filesizes and wrong md5sums with *some* files then. See details in above posting. Actually the new deltup version assumes bzip2 is version 1.0.3 and bzip2_old is version 1.0.2 since these versions are used on the servers and files that have to be compressed with bzip-1.0.2 because version 1.0.3 would break md5sum and filesize are tagged to be compressed with bzip2_old

best wishes,
bp

[blackpenguin]

I think there's a major flaw in the server design...

I am trying to download the updated gettext. It couldn't find any dtu files from any of the servers they've got housing those. So then it'll try to download the full file instead from a different server.

Before when it did this it would automatically start using my "GENTOO_MIRRORS" from make.conf. Now, it sat there for a good 5 minutes trying to download from a server in Russia (I'm in Seattle). All for a 7,550 byte file.

Why does it do this? Doesn't it make more sense to have it go to your "GENTOO_MIRRORS" in this case?

edit:
In fact, upon closer inspection I noticed that in some cases it never tried to download using my GENTOO_MIRRORS at all.

I'm sorry - but this has *nothing* to do with the dynamic-deltup-server - it is PORTAGE-design your are speaking of. getdelta.sh uses the URL portage gives to it when falling back to download the full file. And portage actually uses your GENTOO_MIRRORS to get files - if that fails, portage takes the URL given in the ebuild itself. In any situation getdelta.sh always contacts linux01.gwdg.de and asks for a dtu-file and if it fails it tries the URL given by portage and that is exactly what portage would have used even without getdelta.
best wishes,
bp

[leighgiles]

blackpenguin - thanks for the reply

I think this is a terrific "brains beats brawn" solution

I'll try your suggestion

leigh

[opensas]

Don't know what happened, but while I was typing the former message, all of the sudden my 30 MB firefox donwload went down to a 900 KB dtu file

(I had to let wget do the job, I guss I must have missed something in the # FETCH="/usr/bin/snarf -b ")

but then I got the following trouble

Code: Select all

>>> Downloading http://distfiles.gentoo.org/distfiles/kdelibs-3.4.0.tar.bz2
Searching for a previously downloaded file in /mnt/xp/gentoo/distfiles

We have following candidates to choose from
kdelibs-3.3.2.tar.bz2

The best of all is ... kdelibs-3.3.2.tar.bz2

Checking if this file is OK.

Trying to download kdelibs-3.3.2.tar.bz2-kdelibs-3.4.0.tar.bz2.dtu

error: HTTP error from server: HTTP/1.1 404 Not Found
The dtu could not be fetched, downloading full file from original URL

http://distfiles.gentoo.org/distfiles/kdelibs-3.4.0.tar.bz2 (16466K)
kdelibs-3.4.0.tar.bz2     [                        ]     123K |    2.14K/s

Exiting on signal 2
chown failed on distfile: kdelibs-3.4.0.tar.bz2bash-2.05b#   

If anybody can tell me what shall I do in these cases.

Saludos

Sas

[blackpenguin]

Code: Select all

The Server creates the dtu-file NOW

TIMEOUT exceeded.

The dtu could not be fetched, downloading full file from original URL

--05:57:41--  http://distfiles.gentoo.org/distfiles/firefox-1.0.3-source.tar.bz2
           => `firefox-1.0.3-source.tar.bz2'
Resolving distfiles.gentoo.org... 216.165.129.135, 156.56.247.195, 140.211.166.134
Connecting to distfiles.gentoo.org[216.165.129.135]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32,784,300 [application/x-tar]

 2% [>                                                      ] 737,036        1.83K/s  ETA 5:21:26

Exiting on signal 2

The dynamic deltup-servers create dtu-files on demand. That means if a dtu could not be provided from the cache, it is created just for you. (and for anyone else who wants that dtu later)
"The Server creates the dtu-file NOW" just tells you that. Actually the TIMEOUT (defined in QUEUETIMEOUT in /etc/deltup/getdelta.rc) exceeded so your client has been told not to wait that long and therefore it began to download the full file. Trying a different deltup-server is not necessary - the servers are connected to each other - you always send your request to linux01.gwdg.de and if this one is busy another server takes your request. Anyway that hasn't be the problem here. You were not waiting in the queue - your dtu has actually been created - only your client didn't wait long enough. However no problem - the dtu has been created anyway - so meanwhile I'm pretty sure it exists now.
best wishes,
bp

[chunderbunny]

The problems with bzip2 and wrong filesizes/md5sum is solved

Just update to deltup-0.4.2-r1 -- this ebuild depends on bzip2-1.0.3 and will additonally install a statically linked bzip2-1.0.2 as /usr/bin/bzip_old

It seems that the best solution would be to make bzip2 slotted, so that you can have multiple versions installed simultaneously, and then have a wrapper script which calls the approprate version of bzip2. It's more the "gentoo way" I think.