How to reduce your download traffic by 75% or more

[ts77]

Hi,

thanks a lot for that great tool again.
I just got an error while running 0.5.3 for the first time:

Code: Select all

>>> emerge (1 of 2) sys-devel/gcc-3.4.1-r2 to /
>>> Downloading http://www.research.ibm.com/trl/projects/security/ssp/gcc3_4/protector-3.4-2.tar.gz
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from
protector-3.3-7.tar.gz
protector-3.3.2-2.tar.gz

The best of all is ... protector-3.3.2-2.tar.gz

Checking if this file is OK.

grep: /digest-*: No such file or directory
/usr/local/bin/getdelta.sh: line 428: [: missing `]'
Trying to download protector-3.3.2-2.tar.gz-protector-3.4-2.tar.gz.dtu
...

Any idea why it can't find the digest?

Edit: I have to add that it seems to work though . And thanks for implementing the configurable queue retry .


TIA,

thomas

[blackpenguin]

Any idea why it can't find the digest?

Uhm... Yes
There is a typo (FILES_DIR instead of FILESDIR) - actually I haven't tested this part of the code.
There is also a missing "]" on line 428 (as your error msg said)

Both is fixed now in 0.5.3.1

bp

[ts77]

Heh, I thought the second error was a followup for the first one.

No problem and thanks for the quick fix .

[blackpenguin]

Another bug appeared in the output when deleting old files.
fixed in version 0.5.3.2 which is available at
http://linux01.gwdg.de/~nlissne/getdelta.sh now.
bp

[Pkunkage]

I have submitted
http://bugs.gentoo.org/show_bug.cgi?id=63864
to patch the xdelta package code slightly. The new ebuild and patch will fix problems xdelta has patching LARGE dtu's or tarballs because it doesn't try to limit memory usage by default.

For example in my system with 500MB of Swap and 256MB of RAM , when deltup is patching OOo tarballs or mozilla which are >200MB , xdelta will just plain run out of RAM and start mem-swapping like a forkbomb taking down X , and some other system services like squid etc.

The new patch sets the default memory cache in xdelta to 100 megs. This can be tweaked by editing /etc/deltup/getdelta.rc and setting XDELTA_MAX_MEGS=200
You can change 200 to whatever megabyte limit is appropriate for your system.

[eNTi]

what about putting an version check and automatic upgrade script into getdelta.sh? you could make a configuration variable like "GETDELTA_AUTOUPDATE". i think that would be great.

[Pkunkage]

I think blackpenguin has very wisely decided to not go in for auto-update.

Ultimately this script should become part of deltup ebuild in official portage or another ebuild which the user can upgrade whever he wants.

Since the script is typically running as root it wouldn't be wise to have auto-update for something like that. Just something which the user can easily emerge from portage is good enough. Besides whats the big deal with wget http://patthto/getdelta.sh ?

[eNTi]

if it's in portage it's okay. i just don't want to have to check for updates/bugfixes all the time. i'd still prefere auto-update. what's the big deal? you can always turn it off?

[eNTi]

Code: Select all

Trying to download gcc-3.4.1.tar.bz2-gcc-3.4.2.tar.bz2.dtu

--15:22:26--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=gcc-3.4.1.tar.bz               
2&want=gcc-3.4.2.tar.bz2&version=0.5.3
           => `deltup.php?have=gcc-3.4.1.tar.bz2&want=gcc-3.4.2.tar.bz2&version=               
0.5.3'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: ftp://134.102.120.44/deltup/gcc-3.4.1.tar.bz2-gcc-3.4.2.tar.bz2.dtu [f               
ollowing]
--15:22:27--  ftp://134.102.120.44/deltup/gcc-3.4.1.tar.bz2-gcc-3.4.2.tar.bz2.dt               
u
           => `gcc-3.4.1.tar.bz2-gcc-3.4.2.tar.bz2.dtu'
Connecting to 134.102.120.44:21... failed: Address family not supported by proto               
col.
Giving up.

what is this supposed to mean? is this dtu simply not avaliable?

[pleusicles]

- a configurable maximum acceptable queue-position

Thanks for implementing this feature, and also for the new version in general!

[pleusicles]

... and just a little cosmetic problem: after printing "The Server creates the dtu-file NOW", the output stays yellow until the next "colored" element such as "Yes, we got the dtu-file..."

[ReXX]

Is this topic still alive?

Right now I've got a problem emerging deltup

Code: Select all

>>> Unpacking source...
>>> Unpacking deltup-0.4.0.tar.gz to /var/tmp/portage/deltup-0.4.0/work
>>> Source unpacked.

>>> Install deltup-0.4.0 into /var/tmp/portage/deltup-0.4.0/image/ category app-portage
gcc deltup.cpp -march=athlon-tbird -O2 -pipe -o deltup -lstdc++ -lz -lbz2 -lssl
In file included from deltup.cpp:25:
file.h: In member function `virtual int Injectable_IStream::read(void*, int)':
file.h:132: error: ISO C++ forbids cast to non-reference type used as lvalue
file.h:142: error: ISO C++ forbids cast to non-reference type used as lvalue
make: *** [deltup] Error 1

!!! ERROR: app-portage/deltup-0.4.0 failed.
!!! Function src_install, Line 35, Exitcode 2
!!! (no error message)

I'd like to test the script, but without deltup I guess it won't work.

[PrakashP]

Take a look on page 1.

[Kampi]

works great for me! thank you.

[MooktaKiNG]

I would (and others too) appreciate a sample config file (an up to date file, showing all the available options) in the website, right next to the .sh script :D

Thank you.

[MooktaKiNG]

Correction

It does have a sample config file :D:D

[MooktaKiNG]

One thing i don't like is the GENTOO_MIRRORS bit.
Why does it need to be empty?

The problem is that when it trys to download the full file, it now takes ages.

I would prefer to have a mirror which is close to me.

[blackpenguin]

One thing i don't like is the GENTOO_MIRRORS bit.
Why does it need to be empty?

I didn't like it, too.

Well, the reason was, the client gives the server a hint where to find the file, and - if GENTOO_MIRRORS is not empty portage always tries GENTOO_MIRRORS first - leading to a wrong hint from getdelta.sh to its server.

Here's the good news - you can set GENTOO_MIRRORS now without breaking things.
There is a minor version-update (0.5.3.3) on the server at
http://linux01.gwdg.de/~nlissne/getdelta.sh

With this new version I introduce another way to find the correct mirror-group even if portage tries to use a gentoo mirror first.

Please update to 0.5.3.3 and set your GENTOO_MIRRORS again.

bp

[MooktaKiNG]

One thing i don't like is the GENTOO_MIRRORS bit.
Why does it need to be empty?

I didn't like it, too.

Well, the reason was, the client gives the server a hint where to find the file, and - if GENTOO_MIRRORS is not empty portage always tries GENTOO_MIRRORS first - leading to a wrong hint from getdelta.sh to its server.

Here's the good news - you can set GENTOO_MIRRORS now without breaking things.
There is a minor version-update (0.5.3.3) on the server at
http://linux01.gwdg.de/~nlissne/getdelta.sh

With this new version I introduce another way to find the correct mirror-group even if portage tries to use a gentoo mirror first.

Please update to 0.5.3.3 and set your GENTOO_MIRRORS again.

bp

Nice.

[Twink]

have to say i really love this tool and i'm looking forward to the next 'big' package i upgrade to see just how well it performs (I had already downloaded the packages for xorg and ooffice before starting to use this).

One thing i think would be nice is statistics for the local machine, surely it would be possible to add these in the future (although i understand there's more important things). Being able to see exactly how much i've saved by using this tool is of great interest.

Keep up the great work

[Niosop]

First of all, great job. As a (very slow) dial-up user, this has the potential to save me literally days of downloading when I want to update.

The only problem I have is: it doesn't work on amd64 (I know it's not supposed to). By using USE="~x86" it will compile fine, but it always errors on applying the patch after downloading the delta file.

Has anyone gotten it to work on amd64?

If not, does anyone know where the problem lies or where I should start looking for it?

Thanks for an awesome program,
Niosop

[asph]

maybe it would be a good idea to check the gcc version, and patch file.h if going to use gcc 3.4.x so that it compiles.. having to fix that manually its a pain in the XXX

[Niosop]

Upon further inspection, deltup seems to be working fine. The error seems to be in xdelta or bdelta (think it's xdelta). Once I get home I'll continue troubleshooting.

[blackpenguin]

Upon further inspection, deltup seems to be working fine. The error seems to be in xdelta or bdelta (think it's xdelta). Once I get home I'll continue troubleshooting.

Since we do not use bdelta (yet) for the files created by the dynamic deltup server - xdelta might be the problem.

[millenium_psyrax]

FANTASTIC!!!!!

Thanks guys, this has always been a major portage gripe of mine, I have never, and never will understand why something like this has not been a priority. It has shit to me to tears downloading another entire version of a package to just upgrade a point release, I go to the gnu ftp server, and I see a patch that fits, but portage takes the NEWBIE way out, and just downloads a fresh copy of the update.

Something like this should be in portage, FULL STOP. In fact, it should have been in the original design document.