How to reduce your download traffic by 75% or more

Message

blackpenguin · Post by **blackpenguin** » Fri Aug 27, 2004 7:59 pm

Announcement: dynamic deltup server goes public beta

Especially users on modem- or ISDN-dialin-lines or users who pay for the traffic-volume of their internet-connection will love this.

Some of you might remember deltup, a nice piece of software to reduce download-size, achieved by transfering the difference to the old version of an archive only. (Thanks to John Whitney, who brought deltup to us, and thanks to Joshua P. MacDonald who wrote the underlying xdelta)

Unfortunately deltup was removed from portage tree some weeks ago, since the number of available delta-files has been small and none of them were up-to-date anymore.

Now, situation has changed. I am happy to present you the "dynamic deltup server", and the getdelta.sh-downloadwrapper-script.

The dynamic-deltup-server holds many up-to-date delta-files (*.dtu) in cache and, that's the great news, it creates new dtu-files on demand.

Just curious how much traffic you can save with this update-method?
Have a look at these stats:

http://linux01.gwdg.de/~nlissne/

This is how it works:
Just emerge deltup and install a small script written in bash as a download-wrapper in your /etc/make.conf using the FETCHCOMMAND variable.
Done.

When you emerge an update to your packages, the download-wrapper examines your DISTDIR if there is an old version of the source-archive and then asks the deltup-server to provide the dtu-file.

The server will either send back the requested dtu-file or it will put you on a waiting queue and starts to create the dtu-file for you (and others who might want the same file after you).

Once the download-wrapper got the dtu-file it automagically creates the new version of the archive. If it fails for any reason to get the dtu-file, it just will fall back to the normal download. So anything happens transparently to portage - there is no need to learn any new commands and no need to additional user intervention.

The behaviour of the download-wrapper can be configured by some variables inside the script - or - for your convenience - you can use the same variables in your /etc/make.conf (useful if you update the script later and do not want to set all variables again). You can chose how verbose the output should be, if you want colorful output, when to timeout a dtu-request, if you want to delete the old version of the source-archive after successfully creating the new and things like that.
So, please, have a look at and edit the script and change the variables to your needs before running it
All variables are documented in there. Variables set in /etc/make.conf have higher priority.

I hope this will be useful (actually it IS useful for some people already)

Enjoy!
blackpenguin (see me on freenode for feedback or questions **UPDATE**: I've opened a channel #dynamic-deltup-server for discussions and questions)

If you are interested in running another dynamic-deltup-server (idea is a network of deltup-servers that know about the cache-contents of the others) please contact me.

ian! · Post by **ian!** » Fri Aug 27, 2004 10:09 pm

Great! Works for me!

deelkar · Post by **deelkar** » Fri Aug 27, 2004 11:31 pm

assuming you have not yet enabled your portage overlay, use these steps to emerge and start using dynamic deltup:

be root for this:

Code: Select all

 echo 'PORTDIR_OVERLAY="/usr/local/portage"'>> /etc/make.conf    # enable Portage overlay
 mkdir -p /usr/local/portage/app-portage/deltup                  # create dir for deltup in overlay 
 cd /usr/local/portage/app-portage/deltup 
 wget http://linux01.gwdg.de/~nlissne/deltup-0.4.0.ebuild        # download deltup ebuild to overlay
 ebuild deltup-0.4.0.ebuild digest                               # fetch package and build digest
 cd /usr/local/bin                      
 wget http://linux01.gwdg.de/~nlissne/getdelta.sh                # get the getdelta.sh script
 chmod 755 getdelta.sh                                           # make executable
 mkdir -p /etc/portage                                           # make portage config dir (for keywords file)
 echo "app-portage/deltup ~x86" >> /etc/portage/package.keywords # add deltup to keywords
 emerge deltup                                                   # emerge deltup
 echo 'FETCHCOMMAND="/usr/local/bin/getdelta.sh \${URI}"' >> /etc/make.conf # enable getdelta.sh as new fetchcommand
 getdelta.sh                                                     # run getdelta.sh once to initialize config (optional)

make sure you look into the /etc/deltup directory to set up getdelta.sh and deltup to your needs. (the default values should do fine though)

done!

asph · Post by **asph** » Sat Aug 28, 2004 12:33 am

the script seems to work fine, but the ebuild its like broken:

Code: Select all

nastassja ~ # emerge deltup-0.4.0.ebuild
Calculating dependencies \!!! aux_get(): ebuild for '/deltup-0.4.0' does not exist at:
!!!            /usr/portage//deltup/deltup-0.4.0.ebuild
emerge: create(): aux_get() error on /deltup-0.4.0; aborting...

ew8home · Post by **ew8home** » Sat Aug 28, 2004 2:15 am

Hi!

Try putting the deltup-0.4.0.ebuild into /usr/local/portage/app-portage/deltup/

along with "ebuild /usr/local/portage/app-portage/deltup/deltup-0.4.0.ebuild digest" a following ACCEPT_KEYWORDS="~x86" emerge deltup will work pretty fine.

tomaw · Post by **tomaw** » Sat Aug 28, 2004 10:04 am

Works very nicely here too. Is the server capable of taking the load of people testing this out? Hopefully Gentoo will incorporate something similar to this officially for the poor slow connection users out there

asph · Post by **asph** » Sat Aug 28, 2004 10:09 am

that worked, thanks

also when downloading packages, sometimes i get a "ls de*" not found (when it seems to search for already downloaded packages, if it can find it it echoes the ls error msg).. except for that works fine

TPC · Post by **TPC** » Sat Aug 28, 2004 10:10 am

In file included from deltup.cpp:25:
file.h: In member function `virtual int Injectable_IStream::read(void*, int)':
file.h:132: error: ISO C++ forbids cast to non-reference type used as lvalue
file.h:142: error: ISO C++ forbids cast to non-reference type used as lvalue
make: *** [deltup] Error 1

I guess the thing needs to be fixed to work with gcc 3.4

PrakashP · Post by **PrakashP** » Sat Aug 28, 2004 1:23 pm

It was easy to fix.

Look into source. There are two instances of

(char*)foo+=bar;

change that to
foo=(char*)foo+bar;

TPC · Post by **TPC** » Sat Aug 28, 2004 2:07 pm

it worked, thanks.

What about resuming downloads? does thing thing support that? What should my RESUMECOMMAND be in make.conf?

Greven · Post by **Greven** » Sat Aug 28, 2004 9:03 pm

Will it work with other downloaders??

AKA Snarf?

Code: Select all

#Snarf
FETCHCOMMAND="/usr/bin/snarf -b \${URI} \${DISTDIR}/\${FILE}"
RESUMECOMMAND="/usr/bin/snarf -rb \${URI} \${DISTDIR}/\${FILE}"

Wiebel · Post by **Wiebel** » Sat Aug 28, 2004 9:32 pm

@Greven not yet tested but should work fine, can you report your experience?
but anyway you must use FETCH= instead of FETCHCOMMAND= because gedelta.sh must be the script emerge will start aas FETCHCOMMAND. FETCH on the other hand advices getdelta.sh itself ti use the proper command.

Vanquirius · Post by **Vanquirius** » Sat Aug 28, 2004 9:38 pm

I installed it and it looks promising. I couldn't put in action just yet (it just fell back to normal downloading), however.

edit: got it to work with gaim-encryption-2.29.tar.gz-gaim-encryption-2.30.tar.gz.dtu. Awesome job!

Greven · Post by **Greven** » Sat Aug 28, 2004 9:54 pm

@Wiebel

Sure.... I just emerge synced so I will be adding deltup on soon.

blackpenguin · Post by **blackpenguin** » Sun Aug 29, 2004 12:34 am

Greven wrote:Will it work with other downloaders??

AKA Snarf?

YES, nothing depends on the output or behaviour of wget - that's the reason why there is a variable named FETCH in the script.

But, do *not* change your RESUMECOMMAND in your make.conf - getdelta.sh *cannot* handle resumes. So if you have already parts of an archive in your $DISTDIR, just delete it or let portage use your RESUMECOMMAND to get the rest.

This is also a trick, to prevent portage using getdelta.sh for a specific download (if ever needed) - just "touch $DISTDIR/$filename" - and portage will use RESUMECOMMAND instead of getdelta.sh

blackpenguin · Post by **blackpenguin** » Sun Aug 29, 2004 12:37 am

TPC wrote:What about resuming downloads? does thing thing support that? What should my RESUMECOMMAND be in make.conf?

Do not change it. getdelta.sh cannot resume.

riksta · Post by **riksta** » Sun Aug 29, 2004 2:28 am

forgive me if i am wrong, but why not just use RSYNC for this?

PrakashP · Post by **PrakashP** » Sun Aug 29, 2004 1:59 pm

I wonder why it doesn't always work:

GOT gnupg-1.2.5.tar.bz2-gnupg-1.2.6.tar.bz2.failed

$
{RED}The server could not build the dtu-file for gnupg-1.2.6.tar.bz2

reason:

sorry, could not get gnupg-1.2.6.tar.bz2

After that the original archive is downloaded instead of the delta.

(edit) Happened to me two more times with other files. Something wrong with the server? Too much of traffic?

There are cases where the delta fails miserably (ie gets way too big), usually with very small files:

pcre-4.4.tar.bz2-pcre-4.5.tar.bz2.dtu 54.4 kB 280.0 B -19797.86% -55434.0 B 2004-08-29 15:30 1

I think here the server software should rather throw away the delte and give a error/info back to download the original file.

skipjack · Post by **skipjack** » Sun Aug 29, 2004 2:26 pm

Hi, blackpenguin ..
Hello World!

yeah public beta .. that's great!

I can say to everyone thta script is very cool. it works fine and fast

I was one of the beta tester before the public test starts

I will say taht tool is great and I think it will help gentoo, because I read sometimes that the distfiles and ebuild of gentoo portage makes to many traffic.

What will say the gentoo developer to this tool?

Happy beta testing

NoUseForAName · Post by **NoUseForAName** » Sun Aug 29, 2004 4:59 pm

Looks like a great thing, but I'm having problems using it

This is what i get:

Code: Select all

richard distfiles # emerge -f =development-sources-2.6.8.1
Calculating dependencies ...done!
>>> emerge (1 of 1) sys-kernel/development-sources-2.6.8.1 to /
>>> Downloading http://gentoo.osuosl.org/distfiles/linux-2.6.8.1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

/usr/local/bin/getdelta.sh: line 305: [: too many arguments
We have following candidates to choose from
links-2.1pre14.tar.bz2
linux-2.4.22.tar.bz2
linux-2.6.3.tar.bz2
linux-2.6.7.tar.bz2

The best of all is ... linux-2.6.7.tar.bz2

Trying to download linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu

--18:55:20--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=linux-2.6.7.tar.bz2&want=linux-2.6.8.1.tar.bz2&version=0.5.2
           => `deltup.php?have=linux-2.6.7.tar.bz2&want=linux-2.6.8.1.tar.bz2&version=0.5.2'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: ftp://134.102.120.44/deltup/linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu [following]
--18:55:23--  ftp://134.102.120.44/deltup/linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu
           => `linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu'
/usr/local/bin/getdelta.sh: line 431: 23152 Aborted                 $FETCH "${DELTUP_SERVER}${query}"
The dtu could not be fetched, downloading full file from original URL

Any idea what could be wrong?

NoUseForAName · Post by **NoUseForAName** » Sun Aug 29, 2004 5:12 pm

Ok, got rid of the first error:

Code: Select all

/usr/local/bin/getdelta.sh: line 305: [: too many arguments

by changing line 305 from

Code: Select all

        if ! [ -z $filelist ]

to

Code: Select all

        if ! [ -z "$filelist" ]

but that doesn't seem to matter to much, I'm still getting the last error, even though the file seems to exist.

NoUseForAName · Post by **NoUseForAName** » Sun Aug 29, 2004 5:29 pm

Sorry for the noise, remerged wget and now it works fine.

Looks like a really useful program. Thanks!

blackpenguin · Post by **blackpenguin** » Sun Aug 29, 2004 5:44 pm

NoUseForAName wrote:Ok, got rid of the first error:
Code: Select all
/usr/local/bin/getdelta.sh: line 305: [: too many arguments 
by changing line 305 from
Code: Select all
        if ! [ -z $filelist ]
to
Code: Select all
        if ! [ -z "$filelist" ]
but that doesn't seem to matter to much, I'm still getting the last error, even though the file seems to exist.

It worked even with the "too many arguments" problem (therefore i didn't fixed it before, since there were other - more severe problems before

), but you've fixed that problem. Thanks - change applied.

blackpenguin · Post by **blackpenguin** » Sun Aug 29, 2004 6:13 pm

PrakashKC wrote:I wonder why it doesn't always work:

GOT gnupg-1.2.5.tar.bz2-gnupg-1.2.6.tar.bz2.failed

$
{RED}The server could not build the dtu-file for gnupg-1.2.6.tar.bz2

reason:

sorry, could not get gnupg-1.2.6.tar.bz2

Well, the reason is given there - the server tried to get gnupg-1.2.6.tar.bz2 but failed. Sometimes this happens because the mirror-type given to the server is "gentoo", but the file was not found on any gentoo mirror known to the server.

Set GENTOO_MIRRORS="" in make.conf to avoid such problems (with this setting, portage always will give the original-server-url to getdelta.sh - and getdelta.sh will fail less in finding the correct mirror-group) - the problem is - once the server failed on all mirrors of the given mirror-group it will not try again (with another mirror-group) until I delete the "failed"-file.
But as long as it downloads the original archive in cases like that, IMHO it's not a severe problem.

PrakashKC wrote: There are cases where the delta fails miserably (ie gets way too big), usually with very small files:

pcre-4.4.tar.bz2-pcre-4.5.tar.bz2.dtu 54.4 kB 280.0 B -19797.86% -55434.0 B 2004-08-29 15:30 1

I think here the server software should rather throw away the delte and give a error/info back to download the original file.

Well, in THIS case (pcre) it probably just failed to see the correct original-size (could happen with local symlinks on the server itself) - 280 Bytes is too small for pcre-4.5.tar.bz2

However, there *are* cases, where the dtu is *really* bigger than the original file - and you are right, it should throw away these dtu.
I have put this on my todo-list - its easy to do that - so it will be done soon.

Thanks for your feedback,
blackpenguin

blackpenguin · Post by **blackpenguin** » Sun Aug 29, 2004 6:22 pm

riksta wrote:forgive me if i am wrong, but why not just use RSYNC for this?

Uhm.... what? I am not sure if I catch your idea. - If you mean to unpack the local old version to somewhere and then rsync with the wanted version - well - that cannot be done with a downloadwrapper since you will fail to get the same MD5SUM again when compressing back to an archive again.

That is, because you do cannot know anything about the parameters and (for gzipped files timestamp) originally used.

Just try it - untar a gzipped file and tar it again - portage will not accept the result, since the MD5SUM will have changed.

That is the great thing about John Whitney's deltup, it reads the needed information from the fileheaders of the destination-file to be able to create a patch that makes a bit-identical destination-archive and therefore pass the md5sum-check.

blackpenguin