Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to reduce your download traffic by 75% or more
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3 ... 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Fri Aug 27, 2004 7:59 pm    Post subject: How to reduce your download traffic by 75% or more Reply with quote

:D Announcement: dynamic deltup server goes public beta :D

Especially users on modem- or ISDN-dialin-lines or users who pay for the traffic-volume of their internet-connection will love this.

Some of you might remember deltup, a nice piece of software to reduce download-size, achieved by transfering the difference to the old version of an archive only. (Thanks to John Whitney, who brought deltup to us, and thanks to Joshua P. MacDonald who wrote the underlying xdelta)

Unfortunately deltup was removed from portage tree some weeks ago, since the number of available delta-files has been small and none of them were up-to-date anymore.

Now, situation has changed. I am happy to present you the "dynamic deltup server", and the getdelta.sh-downloadwrapper-script.

The dynamic-deltup-server holds many up-to-date delta-files (*.dtu) in cache and, that's the great news, it creates new dtu-files on demand.

Just curious how much traffic you can save with this update-method?
Have a look at these stats:
:arrow: http://linux01.gwdg.de/~nlissne/

This is how it works:
Just emerge deltup and install a small script written in bash as a download-wrapper in your /etc/make.conf using the FETCHCOMMAND variable.
Done. :)

When you emerge an update to your packages, the download-wrapper examines your DISTDIR if there is an old version of the source-archive and then asks the deltup-server to provide the dtu-file.

The server will either send back the requested dtu-file or it will put you on a waiting queue and starts to create the dtu-file for you (and others who might want the same file after you).

Once the download-wrapper got the dtu-file it automagically creates the new version of the archive. If it fails for any reason to get the dtu-file, it just will fall back to the normal download. So anything happens transparently to portage - there is no need to learn any new commands and no need to additional user intervention.

The behaviour of the download-wrapper can be configured by some variables inside the script - or - for your convenience - you can use the same variables in your /etc/make.conf (useful if you update the script later and do not want to set all variables again). You can chose how verbose the output should be, if you want colorful output, when to timeout a dtu-request, if you want to delete the old version of the source-archive after successfully creating the new and things like that.
So, please, have a look at and edit the script and change the variables to your needs before running it
All variables are documented in there. Variables set in /etc/make.conf have higher priority.

I hope this will be useful (actually it IS useful for some people already)

Enjoy!
blackpenguin (see me on freenode for feedback or questions **UPDATE**: I've opened a channel #dynamic-deltup-server for discussions and questions)

If you are interested in running another dynamic-deltup-server (idea is a network of deltup-servers that know about the cache-contents of the others) please contact me.


Last edited by blackpenguin on Sun Aug 29, 2004 12:28 am; edited 1 time in total
Back to top
View user's profile Send private message
ian!
Bodhisattva
Bodhisattva


Joined: 25 Feb 2003
Posts: 3827
Location: Essen, Germany

PostPosted: Fri Aug 27, 2004 10:09 pm    Post subject: Reply with quote

Great! Works for me!
_________________
"To have a successful open source project, you need to be at least somewhat successful at getting along with people." -- Daniel Robbins
Back to top
View user's profile Send private message
deelkar
n00b
n00b


Joined: 27 Aug 2004
Posts: 6

PostPosted: Fri Aug 27, 2004 11:31 pm    Post subject: Reply with quote

assuming you have not yet enabled your portage overlay, use these steps to emerge and start using dynamic deltup:

be root for this:
Code:

 echo 'PORTDIR_OVERLAY="/usr/local/portage"'>> /etc/make.conf    # enable Portage overlay
 mkdir -p /usr/local/portage/app-portage/deltup                  # create dir for deltup in overlay
 cd /usr/local/portage/app-portage/deltup
 wget http://linux01.gwdg.de/~nlissne/deltup-0.4.0.ebuild        # download deltup ebuild to overlay
 ebuild deltup-0.4.0.ebuild digest                               # fetch package and build digest
 cd /usr/local/bin                     
 wget http://linux01.gwdg.de/~nlissne/getdelta.sh                # get the getdelta.sh script
 chmod 755 getdelta.sh                                           # make executable
 mkdir -p /etc/portage                                           # make portage config dir (for keywords file)
 echo "app-portage/deltup ~x86" >> /etc/portage/package.keywords # add deltup to keywords
 emerge deltup                                                   # emerge deltup
 echo 'FETCHCOMMAND="/usr/local/bin/getdelta.sh \${URI}"' >> /etc/make.conf # enable getdelta.sh as new fetchcommand
 getdelta.sh                                                     # run getdelta.sh once to initialize config (optional)


make sure you look into the /etc/deltup directory to set up getdelta.sh and deltup to your needs. (the default values should do fine though)

done!


Last edited by deelkar on Tue Sep 21, 2004 3:50 am; edited 6 times in total
Back to top
View user's profile Send private message
asph
l33t
l33t


Joined: 25 Aug 2003
Posts: 737
Location: Barcelona, Spain

PostPosted: Sat Aug 28, 2004 12:33 am    Post subject: Reply with quote

the script seems to work fine, but the ebuild its like broken:
Code:
nastassja ~ # emerge deltup-0.4.0.ebuild
Calculating dependencies \!!! aux_get(): ebuild for '/deltup-0.4.0' does not exist at:
!!!            /usr/portage//deltup/deltup-0.4.0.ebuild
emerge: create(): aux_get() error on /deltup-0.4.0; aborting...

_________________
gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep
Back to top
View user's profile Send private message
ew8home
n00b
n00b


Joined: 17 Feb 2004
Posts: 4

PostPosted: Sat Aug 28, 2004 2:15 am    Post subject: Reply with quote

Hi!

Try putting the deltup-0.4.0.ebuild into /usr/local/portage/app-portage/deltup/

along with "ebuild /usr/local/portage/app-portage/deltup/deltup-0.4.0.ebuild digest" a following ACCEPT_KEYWORDS="~x86" emerge deltup will work pretty fine.
Back to top
View user's profile Send private message
tomaw
Guru
Guru


Joined: 26 Mar 2003
Posts: 429
Location: UK

PostPosted: Sat Aug 28, 2004 10:04 am    Post subject: Reply with quote

Works very nicely here too. Is the server capable of taking the load of people testing this out? Hopefully Gentoo will incorporate something similar to this officially for the poor slow connection users out there :)
_________________
Tom Wesley
Back to top
View user's profile Send private message
asph
l33t
l33t


Joined: 25 Aug 2003
Posts: 737
Location: Barcelona, Spain

PostPosted: Sat Aug 28, 2004 10:09 am    Post subject: Reply with quote

that worked, thanks :P

also when downloading packages, sometimes i get a "ls de*" not found (when it seems to search for already downloaded packages, if it can find it it echoes the ls error msg).. except for that works fine
_________________
gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep
Back to top
View user's profile Send private message
TPC
Tux's lil' helper
Tux's lil' helper


Joined: 16 Sep 2003
Posts: 135
Location: Sweden

PostPosted: Sat Aug 28, 2004 10:10 am    Post subject: Reply with quote

Quote:

In file included from deltup.cpp:25:
file.h: In member function `virtual int Injectable_IStream::read(void*, int)':
file.h:132: error: ISO C++ forbids cast to non-reference type used as lvalue
file.h:142: error: ISO C++ forbids cast to non-reference type used as lvalue
make: *** [deltup] Error 1


I guess the thing needs to be fixed to work with gcc 3.4 :roll:
Back to top
View user's profile Send private message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Sat Aug 28, 2004 1:23 pm    Post subject: Reply with quote

It was easy to fix.

Look into source. There are two instances of

(char*)foo+=bar;

change that to
foo=(char*)foo+bar;
Back to top
View user's profile Send private message
TPC
Tux's lil' helper
Tux's lil' helper


Joined: 16 Sep 2003
Posts: 135
Location: Sweden

PostPosted: Sat Aug 28, 2004 2:07 pm    Post subject: Reply with quote

it worked, thanks.

What about resuming downloads? does thing thing support that? What should my RESUMECOMMAND be in make.conf?
Back to top
View user's profile Send private message
Greven
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jul 2002
Posts: 138

PostPosted: Sat Aug 28, 2004 9:03 pm    Post subject: Reply with quote

Will it work with other downloaders??

AKA Snarf?

Code:

#Snarf
FETCHCOMMAND="/usr/bin/snarf -b \${URI} \${DISTDIR}/\${FILE}"
RESUMECOMMAND="/usr/bin/snarf -rb \${URI} \${DISTDIR}/\${FILE}"

_________________
veritas vos liberabit...
Linux User Number: 346805
Wine-Wiki
AMD 64 3500+ | MSI "K8T NEO2-FIR" | mushkin Dual Channel DDR 400
Back to top
View user's profile Send private message
Wiebel
n00b
n00b


Joined: 05 Feb 2003
Posts: 59
Location: Bremen, Germany

PostPosted: Sat Aug 28, 2004 9:32 pm    Post subject: Reply with quote

@Greven not yet tested but should work fine, can you report your experience?
but anyway you must use FETCH= instead of FETCHCOMMAND= because gedelta.sh must be the script emerge will start aas FETCHCOMMAND. FETCH on the other hand advices getdelta.sh itself ti use the proper command.
Back to top
View user's profile Send private message
Vanquirius
Retired Dev
Retired Dev


Joined: 14 Jun 2002
Posts: 1297
Location: Ethereal plains

PostPosted: Sat Aug 28, 2004 9:38 pm    Post subject: Reply with quote

I installed it and it looks promising. I couldn't put in action just yet (it just fell back to normal downloading), however.

edit: got it to work with gaim-encryption-2.29.tar.gz-gaim-encryption-2.30.tar.gz.dtu. Awesome job!
_________________
Hello.
Back to top
View user's profile Send private message
Greven
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jul 2002
Posts: 138

PostPosted: Sat Aug 28, 2004 9:54 pm    Post subject: Reply with quote

@Wiebel

Sure.... I just emerge synced so I will be adding deltup on soon.
_________________
veritas vos liberabit...
Linux User Number: 346805
Wine-Wiki
AMD 64 3500+ | MSI "K8T NEO2-FIR" | mushkin Dual Channel DDR 400
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Aug 29, 2004 12:34 am    Post subject: Reply with quote

Greven wrote:
Will it work with other downloaders??

AKA Snarf?


YES, nothing depends on the output or behaviour of wget - that's the reason why there is a variable named FETCH in the script.

But, do *not* change your RESUMECOMMAND in your make.conf - getdelta.sh *cannot* handle resumes. So if you have already parts of an archive in your $DISTDIR, just delete it or let portage use your RESUMECOMMAND to get the rest.

This is also a trick, to prevent portage using getdelta.sh for a specific download (if ever needed) - just "touch $DISTDIR/$filename" - and portage will use RESUMECOMMAND instead of getdelta.sh
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Aug 29, 2004 12:37 am    Post subject: Reply with quote

TPC wrote:
What about resuming downloads? does thing thing support that? What should my RESUMECOMMAND be in make.conf?


Do not change it. getdelta.sh cannot resume.
Back to top
View user's profile Send private message
riksta
n00b
n00b


Joined: 16 Apr 2004
Posts: 73
Location: Manchester, UK

PostPosted: Sun Aug 29, 2004 2:28 am    Post subject: Reply with quote

forgive me if i am wrong, but why not just use RSYNC for this?
Back to top
View user's profile Send private message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Sun Aug 29, 2004 1:59 pm    Post subject: Reply with quote

I wonder why it doesn't always work:

GOT gnupg-1.2.5.tar.bz2-gnupg-1.2.6.tar.bz2.failed

$
{RED}The server could not build the dtu-file for gnupg-1.2.6.tar.bz2

reason:

sorry, could not get gnupg-1.2.6.tar.bz2


After that the original archive is downloaded instead of the delta.

(edit) Happened to me two more times with other files. Something wrong with the server? Too much of traffic?

There are cases where the delta fails miserably (ie gets way too big), usually with very small files:


pcre-4.4.tar.bz2-pcre-4.5.tar.bz2.dtu 54.4 kB 280.0 B -19797.86% -55434.0 B 2004-08-29 15:30 1

I think here the server software should rather throw away the delte and give a error/info back to download the original file.


Last edited by PrakashP on Sun Aug 29, 2004 2:33 pm; edited 3 times in total
Back to top
View user's profile Send private message
skipjack
Tux's lil' helper
Tux's lil' helper


Joined: 20 Mar 2003
Posts: 75

PostPosted: Sun Aug 29, 2004 2:26 pm    Post subject: Reply with quote

Hi, blackpenguin ..
Hello World! ;-)

yeah public beta .. that's great!

I can say to everyone thta script is very cool. it works fine and fast ;-)
I was one of the beta tester before the public test starts ;-)
I will say taht tool is great and I think it will help gentoo, because I read sometimes that the distfiles and ebuild of gentoo portage makes to many traffic.

What will say the gentoo developer to this tool?

Happy beta testing ;-)
Back to top
View user's profile Send private message
NoUseForAName
Tux's lil' helper
Tux's lil' helper


Joined: 22 Feb 2003
Posts: 84
Location: Sweden

PostPosted: Sun Aug 29, 2004 4:59 pm    Post subject: Reply with quote

Looks like a great thing, but I'm having problems using it :(

This is what i get:

Code:
richard distfiles # emerge -f =development-sources-2.6.8.1
Calculating dependencies ...done!
>>> emerge (1 of 1) sys-kernel/development-sources-2.6.8.1 to /
>>> Downloading http://gentoo.osuosl.org/distfiles/linux-2.6.8.1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

/usr/local/bin/getdelta.sh: line 305: [: too many arguments
We have following candidates to choose from
links-2.1pre14.tar.bz2
linux-2.4.22.tar.bz2
linux-2.6.3.tar.bz2
linux-2.6.7.tar.bz2

The best of all is ... linux-2.6.7.tar.bz2

Trying to download linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu

--18:55:20--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=linux-2.6.7.tar.bz2&want=linux-2.6.8.1.tar.bz2&version=0.5.2
           => `deltup.php?have=linux-2.6.7.tar.bz2&want=linux-2.6.8.1.tar.bz2&version=0.5.2'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: ftp://134.102.120.44/deltup/linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu [following]
--18:55:23--  ftp://134.102.120.44/deltup/linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu
           => `linux-2.6.7.tar.bz2-linux-2.6.8.1.tar.bz2.dtu'
/usr/local/bin/getdelta.sh: line 431: 23152 Aborted                 $FETCH "${DELTUP_SERVER}${query}"
The dtu could not be fetched, downloading full file from original URL


Any idea what could be wrong?
Back to top
View user's profile Send private message
NoUseForAName
Tux's lil' helper
Tux's lil' helper


Joined: 22 Feb 2003
Posts: 84
Location: Sweden

PostPosted: Sun Aug 29, 2004 5:12 pm    Post subject: Reply with quote

Ok, got rid of the first error:
Code:
/usr/local/bin/getdelta.sh: line 305: [: too many arguments


by changing line 305 from
Code:
        if ! [ -z $filelist ]

to
Code:
        if ! [ -z "$filelist" ]


but that doesn't seem to matter to much, I'm still getting the last error, even though the file seems to exist.
Back to top
View user's profile Send private message
NoUseForAName
Tux's lil' helper
Tux's lil' helper


Joined: 22 Feb 2003
Posts: 84
Location: Sweden

PostPosted: Sun Aug 29, 2004 5:29 pm    Post subject: Reply with quote

Sorry for the noise, remerged wget and now it works fine.

Looks like a really useful program. Thanks!
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Aug 29, 2004 5:44 pm    Post subject: Reply with quote

NoUseForAName wrote:
Ok, got rid of the first error:
Code:
/usr/local/bin/getdelta.sh: line 305: [: too many arguments


by changing line 305 from
Code:
        if ! [ -z $filelist ]

to
Code:
        if ! [ -z "$filelist" ]


but that doesn't seem to matter to much, I'm still getting the last error, even though the file seems to exist.


It worked even with the "too many arguments" problem (therefore i didn't fixed it before, since there were other - more severe problems before ;-)), but you've fixed that problem. Thanks - change applied.
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Aug 29, 2004 6:13 pm    Post subject: Reply with quote

PrakashKC wrote:
I wonder why it doesn't always work:

GOT gnupg-1.2.5.tar.bz2-gnupg-1.2.6.tar.bz2.failed

$
{RED}The server could not build the dtu-file for gnupg-1.2.6.tar.bz2

reason:

sorry, could not get gnupg-1.2.6.tar.bz2


Well, the reason is given there - the server tried to get gnupg-1.2.6.tar.bz2 but failed. Sometimes this happens because the mirror-type given to the server is "gentoo", but the file was not found on any gentoo mirror known to the server.

Set GENTOO_MIRRORS="" in make.conf to avoid such problems (with this setting, portage always will give the original-server-url to getdelta.sh - and getdelta.sh will fail less in finding the correct mirror-group) - the problem is - once the server failed on all mirrors of the given mirror-group it will not try again (with another mirror-group) until I delete the "failed"-file.
But as long as it downloads the original archive in cases like that, IMHO it's not a severe problem.

PrakashKC wrote:

There are cases where the delta fails miserably (ie gets way too big), usually with very small files:

pcre-4.4.tar.bz2-pcre-4.5.tar.bz2.dtu 54.4 kB 280.0 B -19797.86% -55434.0 B 2004-08-29 15:30 1

I think here the server software should rather throw away the delte and give a error/info back to download the original file.


Well, in THIS case (pcre) it probably just failed to see the correct original-size (could happen with local symlinks on the server itself) - 280 Bytes is too small for pcre-4.5.tar.bz2

However, there *are* cases, where the dtu is *really* bigger than the original file - and you are right, it should throw away these dtu.
I have put this on my todo-list - its easy to do that - so it will be done soon.

Thanks for your feedback,
blackpenguin


Last edited by blackpenguin on Sun Aug 29, 2004 7:12 pm; edited 1 time in total
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Aug 29, 2004 6:22 pm    Post subject: Reply with quote

riksta wrote:
forgive me if i am wrong, but why not just use RSYNC for this?


Uhm.... what? I am not sure if I catch your idea. - If you mean to unpack the local old version to somewhere and then rsync with the wanted version - well - that cannot be done with a downloadwrapper since you will fail to get the same MD5SUM again when compressing back to an archive again.

That is, because you do cannot know anything about the parameters and (for gzipped files timestamp) originally used.

Just try it - untar a gzipped file and tar it again - portage will not accept the result, since the MD5SUM will have changed.

That is the great thing about John Whitney's deltup, it reads the needed information from the fileheaders of the destination-file to be able to create a patch that makes a bit-identical destination-archive and therefore pass the md5sum-check.

blackpenguin
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2, 3 ... 9, 10, 11  Next
Page 1 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum