View previous topic :: View next topic |
Author |
Message |
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Fri Aug 27, 2004 7:59 pm Post subject: How to reduce your download traffic by 75% or more |
|
|
Announcement: dynamic deltup server goes public beta
Especially users on modem- or ISDN-dialin-lines or users who pay for the traffic-volume of their internet-connection will love this.
Some of you might remember deltup, a nice piece of software to reduce download-size, achieved by transfering the difference to the old version of an archive only. (Thanks to John Whitney, who brought deltup to us, and thanks to Joshua P. MacDonald who wrote the underlying xdelta)
Unfortunately deltup was removed from portage tree some weeks ago, since the number of available delta-files has been small and none of them were up-to-date anymore.
Now, situation has changed. I am happy to present you the "dynamic deltup server", and the getdelta.sh-downloadwrapper-script.
The dynamic-deltup-server holds many up-to-date delta-files (*.dtu) in cache and, that's the great news, it creates new dtu-files on demand.
Just curious how much traffic you can save with this update-method?
Have a look at these stats:
http://linux01.gwdg.de/~nlissne/
This is how it works:
Just emerge deltup and install a small script written in bash as a download-wrapper in your /etc/make.conf using the FETCHCOMMAND variable.
Done.
When you emerge an update to your packages, the download-wrapper examines your DISTDIR if there is an old version of the source-archive and then asks the deltup-server to provide the dtu-file.
The server will either send back the requested dtu-file or it will put you on a waiting queue and starts to create the dtu-file for you (and others who might want the same file after you).
Once the download-wrapper got the dtu-file it automagically creates the new version of the archive. If it fails for any reason to get the dtu-file, it just will fall back to the normal download. So anything happens transparently to portage - there is no need to learn any new commands and no need to additional user intervention.
The behaviour of the download-wrapper can be configured by some variables inside the script - or - for your convenience - you can use the same variables in your /etc/make.conf (useful if you update the script later and do not want to set all variables again). You can chose how verbose the output should be, if you want colorful output, when to timeout a dtu-request, if you want to delete the old version of the source-archive after successfully creating the new and things like that.
So, please, have a look at and edit the script and change the variables to your needs before running it
All variables are documented in there. Variables set in /etc/make.conf have higher priority.
I hope this will be useful (actually it IS useful for some people already)
Enjoy!
blackpenguin (see me on freenode for feedback or questions **UPDATE**: I've opened a channel #dynamic-deltup-server for discussions and questions)
If you are interested in running another dynamic-deltup-server (idea is a network of deltup-servers that know about the cache-contents of the others) please contact me.
Last edited by blackpenguin on Sun Aug 29, 2004 12:28 am; edited 1 time in total |
|
Back to top |
|
|
ian! Bodhisattva
Joined: 25 Feb 2003 Posts: 3829 Location: Essen, Germany
|
Posted: Fri Aug 27, 2004 10:09 pm Post subject: |
|
|
Great! Works for me! _________________ "To have a successful open source project, you need to be at least somewhat successful at getting along with people." -- Daniel Robbins |
|
Back to top |
|
|
deelkar n00b
Joined: 27 Aug 2004 Posts: 8
|
Posted: Fri Aug 27, 2004 11:31 pm Post subject: |
|
|
assuming you have not yet enabled your portage overlay, use these steps to emerge and start using dynamic deltup:
be root for this: Code: |
echo 'PORTDIR_OVERLAY="/usr/local/portage"'>> /etc/make.conf # enable Portage overlay
mkdir -p /usr/local/portage/app-portage/deltup # create dir for deltup in overlay
cd /usr/local/portage/app-portage/deltup
wget http://linux01.gwdg.de/~nlissne/deltup-0.4.0.ebuild # download deltup ebuild to overlay
ebuild deltup-0.4.0.ebuild digest # fetch package and build digest
cd /usr/local/bin
wget http://linux01.gwdg.de/~nlissne/getdelta.sh # get the getdelta.sh script
chmod 755 getdelta.sh # make executable
mkdir -p /etc/portage # make portage config dir (for keywords file)
echo "app-portage/deltup ~x86" >> /etc/portage/package.keywords # add deltup to keywords
emerge deltup # emerge deltup
echo 'FETCHCOMMAND="/usr/local/bin/getdelta.sh \${URI}"' >> /etc/make.conf # enable getdelta.sh as new fetchcommand
getdelta.sh # run getdelta.sh once to initialize config (optional)
|
make sure you look into the /etc/deltup directory to set up getdelta.sh and deltup to your needs. (the default values should do fine though)
done!
Last edited by deelkar on Tue Sep 21, 2004 3:50 am; edited 6 times in total |
|
Back to top |
|
|
asph l33t
Joined: 25 Aug 2003 Posts: 741 Location: Barcelona, Spain
|
Posted: Sat Aug 28, 2004 12:33 am Post subject: |
|
|
the script seems to work fine, but the ebuild its like broken:
Code: | nastassja ~ # emerge deltup-0.4.0.ebuild
Calculating dependencies \!!! aux_get(): ebuild for '/deltup-0.4.0' does not exist at:
!!! /usr/portage//deltup/deltup-0.4.0.ebuild
emerge: create(): aux_get() error on /deltup-0.4.0; aborting...
|
_________________ gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep |
|
Back to top |
|
|
ew8home n00b
Joined: 17 Feb 2004 Posts: 4
|
Posted: Sat Aug 28, 2004 2:15 am Post subject: |
|
|
Hi!
Try putting the deltup-0.4.0.ebuild into /usr/local/portage/app-portage/deltup/
along with "ebuild /usr/local/portage/app-portage/deltup/deltup-0.4.0.ebuild digest" a following ACCEPT_KEYWORDS="~x86" emerge deltup will work pretty fine. |
|
Back to top |
|
|
tomaw Guru
Joined: 26 Mar 2003 Posts: 429 Location: UK
|
Posted: Sat Aug 28, 2004 10:04 am Post subject: |
|
|
Works very nicely here too. Is the server capable of taking the load of people testing this out? Hopefully Gentoo will incorporate something similar to this officially for the poor slow connection users out there _________________ Tom Wesley |
|
Back to top |
|
|
asph l33t
Joined: 25 Aug 2003 Posts: 741 Location: Barcelona, Spain
|
Posted: Sat Aug 28, 2004 10:09 am Post subject: |
|
|
that worked, thanks
also when downloading packages, sometimes i get a "ls de*" not found (when it seems to search for already downloaded packages, if it can find it it echoes the ls error msg).. except for that works fine _________________ gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep |
|
Back to top |
|
|
TPC Tux's lil' helper
Joined: 16 Sep 2003 Posts: 135 Location: Sweden
|
Posted: Sat Aug 28, 2004 10:10 am Post subject: |
|
|
Quote: |
In file included from deltup.cpp:25:
file.h: In member function `virtual int Injectable_IStream::read(void*, int)':
file.h:132: error: ISO C++ forbids cast to non-reference type used as lvalue
file.h:142: error: ISO C++ forbids cast to non-reference type used as lvalue
make: *** [deltup] Error 1
|
I guess the thing needs to be fixed to work with gcc 3.4 |
|
Back to top |
|
|
PrakashP Veteran
Joined: 27 Oct 2003 Posts: 1249 Location: C.C.A.A., Germania
|
Posted: Sat Aug 28, 2004 1:23 pm Post subject: |
|
|
It was easy to fix.
Look into source. There are two instances of
(char*)foo+=bar;
change that to
foo=(char*)foo+bar; |
|
Back to top |
|
|
TPC Tux's lil' helper
Joined: 16 Sep 2003 Posts: 135 Location: Sweden
|
Posted: Sat Aug 28, 2004 2:07 pm Post subject: |
|
|
it worked, thanks.
What about resuming downloads? does thing thing support that? What should my RESUMECOMMAND be in make.conf? |
|
Back to top |
|
|
Greven Tux's lil' helper
Joined: 28 Jul 2002 Posts: 138
|
Posted: Sat Aug 28, 2004 9:03 pm Post subject: |
|
|
Will it work with other downloaders??
AKA Snarf?
Code: |
#Snarf
FETCHCOMMAND="/usr/bin/snarf -b \${URI} \${DISTDIR}/\${FILE}"
RESUMECOMMAND="/usr/bin/snarf -rb \${URI} \${DISTDIR}/\${FILE}"
|
_________________ veritas vos liberabit...
Linux User Number: 346805
Wine-Wiki
AMD 64 3500+ | MSI "K8T NEO2-FIR" | mushkin Dual Channel DDR 400 |
|
Back to top |
|
|
Wiebel n00b
Joined: 05 Feb 2003 Posts: 60 Location: Bad Homburg, Germany
|
Posted: Sat Aug 28, 2004 9:32 pm Post subject: |
|
|
@Greven not yet tested but should work fine, can you report your experience?
but anyway you must use FETCH= instead of FETCHCOMMAND= because gedelta.sh must be the script emerge will start aas FETCHCOMMAND. FETCH on the other hand advices getdelta.sh itself ti use the proper command. |
|
Back to top |
|
|
Vanquirius Retired Dev
Joined: 14 Jun 2002 Posts: 1297 Location: Ethereal plains
|
Posted: Sat Aug 28, 2004 9:38 pm Post subject: |
|
|
I installed it and it looks promising. I couldn't put in action just yet (it just fell back to normal downloading), however.
edit: got it to work with gaim-encryption-2.29.tar.gz-gaim-encryption-2.30.tar.gz.dtu. Awesome job! _________________ Hello. |
|
Back to top |
|
|
Greven Tux's lil' helper
Joined: 28 Jul 2002 Posts: 138
|
Posted: Sat Aug 28, 2004 9:54 pm Post subject: |
|
|
@Wiebel
Sure.... I just emerge synced so I will be adding deltup on soon. _________________ veritas vos liberabit...
Linux User Number: 346805
Wine-Wiki
AMD 64 3500+ | MSI "K8T NEO2-FIR" | mushkin Dual Channel DDR 400 |
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Sun Aug 29, 2004 12:34 am Post subject: |
|
|
Greven wrote: | Will it work with other downloaders??
AKA Snarf? |
YES, nothing depends on the output or behaviour of wget - that's the reason why there is a variable named FETCH in the script.
But, do *not* change your RESUMECOMMAND in your make.conf - getdelta.sh *cannot* handle resumes. So if you have already parts of an archive in your $DISTDIR, just delete it or let portage use your RESUMECOMMAND to get the rest.
This is also a trick, to prevent portage using getdelta.sh for a specific download (if ever needed) - just "touch $DISTDIR/$filename" - and portage will use RESUMECOMMAND instead of getdelta.sh |
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Sun Aug 29, 2004 12:37 am Post subject: |
|
|
TPC wrote: | What about resuming downloads? does thing thing support that? What should my RESUMECOMMAND be in make.conf? |
Do not change it. getdelta.sh cannot resume. |
|
Back to top |
|
|
riksta n00b
Joined: 16 Apr 2004 Posts: 73 Location: Manchester, UK
|
Posted: Sun Aug 29, 2004 2:28 am Post subject: |
|
|
forgive me if i am wrong, but why not just use RSYNC for this? |
|
Back to top |
|
|
PrakashP Veteran
Joined: 27 Oct 2003 Posts: 1249 Location: C.C.A.A., Germania
|
Posted: Sun Aug 29, 2004 1:59 pm Post subject: |
|
|
I wonder why it doesn't always work:
GOT gnupg-1.2.5.tar.bz2-gnupg-1.2.6.tar.bz2.failed
$
{RED}The server could not build the dtu-file for gnupg-1.2.6.tar.bz2
reason:
sorry, could not get gnupg-1.2.6.tar.bz2
After that the original archive is downloaded instead of the delta.
(edit) Happened to me two more times with other files. Something wrong with the server? Too much of traffic?
There are cases where the delta fails miserably (ie gets way too big), usually with very small files:
pcre-4.4.tar.bz2-pcre-4.5.tar.bz2.dtu 54.4 kB 280.0 B -19797.86% -55434.0 B 2004-08-29 15:30 1
I think here the server software should rather throw away the delte and give a error/info back to download the original file.
Last edited by PrakashP on Sun Aug 29, 2004 2:33 pm; edited 3 times in total |
|
Back to top |
|
|
skipjack Tux's lil' helper
Joined: 20 Mar 2003 Posts: 75
|
Posted: Sun Aug 29, 2004 2:26 pm Post subject: |
|
|
Hi, blackpenguin ..
Hello World!
yeah public beta .. that's great!
I can say to everyone thta script is very cool. it works fine and fast
I was one of the beta tester before the public test starts
I will say taht tool is great and I think it will help gentoo, because I read sometimes that the distfiles and ebuild of gentoo portage makes to many traffic.
What will say the gentoo developer to this tool?
Happy beta testing |
|
Back to top |
|
|
NoUseForAName Tux's lil' helper
Joined: 22 Feb 2003 Posts: 84 Location: Sweden
|
Posted: Sun Aug 29, 2004 4:59 pm Post subject: |
|
|
Looks like a great thing, but I'm having problems using it
This is what i get:
Code: | richard distfiles # emerge -f =development-sources-2.6.8.1
Calculating dependencies ...done!
>>> emerge (1 of 1) sys-kernel/development-sources-2.6.8.1 to /
>>> Downloading http://gentoo.osuosl.org/distfiles/linux-2.6.8.1.tar.bz2
Checking if a Thirdparty-Mirror is used ...
No third-party mirror.
Searching for a previosly downloaded file in /usr/portage/distfiles
/usr/local/bin/getdelta.sh: line 305: [: too many arguments
We have following candidates to choose from
links-2.1pre14.tar.bz2
linux-2.4.22.tar.bz2
linux-2.6.3.tar.bz2
linux-2.6.7.tar.bz2
The best of all is ... linux-2.6.7.tar.bz2
Trying to download linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu
--18:55:20-- http://linux01.gwdg.de/%7Enlissne/deltup.php?have=linux-2.6.7.tar.bz2&want=linux-2.6.8.1.tar.bz2&version=0.5.2
=> `deltup.php?have=linux-2.6.7.tar.bz2&want=linux-2.6.8.1.tar.bz2&version=0.5.2'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: ftp://134.102.120.44/deltup/linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu [following]
--18:55:23-- ftp://134.102.120.44/deltup/linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu
=> `linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu'
/usr/local/bin/getdelta.sh: line 431: 23152 Aborted $FETCH "${DELTUP_SERVER}${query}"
The dtu could not be fetched, downloading full file from original URL
|
Any idea what could be wrong? |
|
Back to top |
|
|
NoUseForAName Tux's lil' helper
Joined: 22 Feb 2003 Posts: 84 Location: Sweden
|
Posted: Sun Aug 29, 2004 5:12 pm Post subject: |
|
|
Ok, got rid of the first error:
Code: | /usr/local/bin/getdelta.sh: line 305: [: too many arguments |
by changing line 305 from
Code: | if ! [ -z $filelist ] |
to
Code: | if ! [ -z "$filelist" ] |
but that doesn't seem to matter to much, I'm still getting the last error, even though the file seems to exist. |
|
Back to top |
|
|
NoUseForAName Tux's lil' helper
Joined: 22 Feb 2003 Posts: 84 Location: Sweden
|
Posted: Sun Aug 29, 2004 5:29 pm Post subject: |
|
|
Sorry for the noise, remerged wget and now it works fine.
Looks like a really useful program. Thanks! |
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Sun Aug 29, 2004 5:44 pm Post subject: |
|
|
NoUseForAName wrote: | Ok, got rid of the first error:
Code: | /usr/local/bin/getdelta.sh: line 305: [: too many arguments |
by changing line 305 from
Code: | if ! [ -z $filelist ] |
to
Code: | if ! [ -z "$filelist" ] |
but that doesn't seem to matter to much, I'm still getting the last error, even though the file seems to exist. |
It worked even with the "too many arguments" problem (therefore i didn't fixed it before, since there were other - more severe problems before ), but you've fixed that problem. Thanks - change applied. |
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Sun Aug 29, 2004 6:13 pm Post subject: |
|
|
PrakashKC wrote: | I wonder why it doesn't always work:
GOT gnupg-1.2.5.tar.bz2-gnupg-1.2.6.tar.bz2.failed
$
{RED}The server could not build the dtu-file for gnupg-1.2.6.tar.bz2
reason:
sorry, could not get gnupg-1.2.6.tar.bz2
|
Well, the reason is given there - the server tried to get gnupg-1.2.6.tar.bz2 but failed. Sometimes this happens because the mirror-type given to the server is "gentoo", but the file was not found on any gentoo mirror known to the server.
Set GENTOO_MIRRORS="" in make.conf to avoid such problems (with this setting, portage always will give the original-server-url to getdelta.sh - and getdelta.sh will fail less in finding the correct mirror-group) - the problem is - once the server failed on all mirrors of the given mirror-group it will not try again (with another mirror-group) until I delete the "failed"-file.
But as long as it downloads the original archive in cases like that, IMHO it's not a severe problem.
PrakashKC wrote: |
There are cases where the delta fails miserably (ie gets way too big), usually with very small files:
pcre-4.4.tar.bz2-pcre-4.5.tar.bz2.dtu 54.4 kB 280.0 B -19797.86% -55434.0 B 2004-08-29 15:30 1
I think here the server software should rather throw away the delte and give a error/info back to download the original file. |
Well, in THIS case (pcre) it probably just failed to see the correct original-size (could happen with local symlinks on the server itself) - 280 Bytes is too small for pcre-4.5.tar.bz2
However, there *are* cases, where the dtu is *really* bigger than the original file - and you are right, it should throw away these dtu.
I have put this on my todo-list - its easy to do that - so it will be done soon.
Thanks for your feedback,
blackpenguin
Last edited by blackpenguin on Sun Aug 29, 2004 7:12 pm; edited 1 time in total |
|
Back to top |
|
|
blackpenguin n00b
Joined: 09 Mar 2004 Posts: 43 Location: Germany
|
Posted: Sun Aug 29, 2004 6:22 pm Post subject: |
|
|
riksta wrote: | forgive me if i am wrong, but why not just use RSYNC for this? |
Uhm.... what? I am not sure if I catch your idea. - If you mean to unpack the local old version to somewhere and then rsync with the wanted version - well - that cannot be done with a downloadwrapper since you will fail to get the same MD5SUM again when compressing back to an archive again.
That is, because you do cannot know anything about the parameters and (for gzipped files timestamp) originally used.
Just try it - untar a gzipped file and tar it again - portage will not accept the result, since the MD5SUM will have changed.
That is the great thing about John Whitney's deltup, it reads the needed information from the fileheaders of the destination-file to be able to create a patch that makes a bit-identical destination-archive and therefore pass the md5sum-check.
blackpenguin |
|
Back to top |
|
|
|