How to reduce your download traffic by 75% or more

Message

blackpenguin · Post by **blackpenguin** » Wed May 11, 2005 10:37 pm

Announcement: getdelta.sh and the dynamic deltup server network can be used with amd64 now, too

This is possible, because we have switched from xdelta to bdelta - and bdelta does not only create smaller patches, it also is perfectly amd64 compatible.

Welcome to all amd64-users who like to use deltup for updating.

Have fun.
blackpenguin

localghost · Post by **localghost** » Thu May 12, 2005 10:49 am

prymitive wrote:
thomasa88 wrote:Sweet prg

(kernel-6.8.2 -> 6.8.11-r8 = 5 MB instead of 36 )
(I had kernel 6.8.11-r6 b4 )
where did You get that stuff?? Did Linus send You a beta of 6.8.11 kernel that he is developing in secret? (or did You ment 2.6.8 -> 2.6.11 but numers are not Your domain )??

He's probably dyscalculic.

Don't attribute to malice what can easily be attributed to stupidity. (Murphy, if I'm correct)

opensas · Post by **opensas** » Sat Jun 04, 2005 6:16 am

Hi blackpenguin, just a suggestion to add to getdelta.rc

When dealing with small files (something between 20k ~ 100k) it's more efficient to just download thefile than going to deltup server, build the diff, donloading, unpacking, patching, etc...

It would be nice to have a setting in getdelta.rc to establish the minimum size of the file to use deltup, if the required file is greater, then just donwload the whole thing. (Something like MINIMUN_TARGET_SIZE)

Optionally, to complete avoid going to the deltup server, there acould be another setting for skiping the file if the previously downloaded file exceeds a certain size, that is we would be assuming that the required file has more or less the same size than the previous one. (Something like MINIMUN_SOURCE_SIZE)

Well, it's just an idea...

Saludos

Sas

PS: Right know I'm upgrading my whole kde 3.4 to 3.4.1... and it's working like a charm.

(Obviously, lots of people have done the same before me, that's what I found every diff file I need)

Without getdelta, I wouldn't have even thought about downloading the whole kde stuff thru the phone line.

Once again, great work blackpenguin

BTW, I just emerged emerge-delta-webrsync. I decided to give it a try. (are you involved in that project too???)

Pseud · Post by **Pseud** » Wed Jun 15, 2005 2:36 pm

I just emerged getdelta, but can't seem to find /etc/deltup/getdelta.rc ???
In fact even the /etc/deltup folder doesnt exist
Something wrong with the ebuild?

Pseud · Post by **Pseud** » Wed Jun 15, 2005 4:14 pm

Oh, heh.
getdelta.rc was created the very next time I tried using emerge

ttuttle · Post by **ttuttle** » Sat Jun 18, 2005 11:39 pm

Would this be useful for broadband users as well? I have DSL, but some packages (OpenOffice, X.org, and so on) are still quite big.

Perhaps, if you added the "minimum file size" option, you could also add an option to choose a good value for it based on connection speed and processor speed. (The idea being that the decision is a balance between using bandwidth to download extra data and using processor time to unpack/repack just the changes.)

localghost · Post by **localghost** » Mon Jun 20, 2005 10:13 am

ThinkingInBinary wrote:Would this be useful for broadband users as well? I have DSL, but some packages (OpenOffice, X.org, and so on) are still quite big.

Perhaps, if you added the "minimum file size" option, you could also add an option to choose a good value for it based on connection speed and processor speed. (The idea being that the decision is a balance between using bandwidth to download extra data and using processor time to unpack/repack just the changes.)

I have cable (4.4 Mbps down, 192 Kbps up), and I use it.
I don't care how long it takes, my compile farm is fassssst enough.

larryOnOff · Post by **larryOnOff** » Tue Jul 19, 2005 6:25 am

Can getdelta download portage-*.tar.bz2?
It will be helpfull, very helpful.

lefou · Post by **lefou** » Tue Jul 19, 2005 6:51 am

You should try app-portage/emerge-delta-webrsync. This is what you want.

lefou

opensas · Post by **opensas** » Tue Jul 26, 2005 4:46 am

I do recommend to at least give a try to emerge-delta-webrsync

The script was a bit buggy, but the whole process is quite simple

I'll give you a step by step example:

first emerge tarsync (so that you can sync the portage tree without having to untar the file

then you need a tar file, like portage-20050722.tar

you can one bzipped tar file get it from http://gentoo.osuosl.org/snapshots/

all you have to do is download it and unzip it

then you donwload the following the delta files files:

Code: Select all

wget -nc http://egret.gentoo.org/~gpatches/snapshot-deltas/snapshot-20050722-20050723.patch.bz2
wget -nc http://egret.gentoo.org/~gpatches/snapshot-deltas/snapshot-20050723-20050724.patch.bz2

and the unziped md5sum

Code: Select all

wget -nc http://egret.gentoo.org/~gpatches/snapshots/portage-20050724.tar.bz2.umd5sum

and run the following

Code: Select all

patcher portage-20050722.tar snapshot-20050722-20050723.patch.bz2 snapshot-20050723-20050724.patch.bz2 portage-20050724.tar

we check the new file

Code: Select all

md5sum -c portage-20050724.tar.bz2.umd5sum

we should get something like

Code: Select all

portage-20050724.tar: OK

if everything works OK, then we can sync our portage tree from the tar file with the following command

Code: Select all

tarsync portage-20050724.tar /usr/portage -v -s 1 -o portage -g portage -e /distfiles -e /packages -e /local

then we issue an

Code: Select all

emerge metadata

and in my case I also issue an

Code: Select all

update-eix

With all this in mind it shouldn't be difficult (for someone with some minimun bash knowledge, wich is unfortunately not my case) to make a script to automatize these tasks

Saludos

Sas

PS: Take into account that every delta file is about 100KB, and it's quite unusual to have some delta bigger than 200KB, and each bzipped tar file is 19MB, so you can get an idea of the download time saving!!!

ferringb · Post by **ferringb** » Tue Jul 26, 2005 4:51 am

opensas wrote:I do recommend to at least give a try to emerge-delta-webrsync
The script was a bit buggy, but the whole process is quite simple

Buggy how?

opensas wrote:With all this in mind it shouldn't be difficult (for someone with some minimun bash knowledge, wich is unfortunately not my case) to make a script to automatize these tasks

This is what emerge-delta-webrsync does already (and it's in bash)

opensas wrote:PS: Take into account that every delta file is about 100KB, and it's quite unusual to have some delta bigger than 200KB, and each bzipped tar file is 19MB, so you can get an idea of the download time saving!!!

Occasionally peaks to 300kB, but yeah, savings are on the order of ~99%

opensas · Post by **opensas** » Tue Jul 26, 2005 7:46 am

Hi ferringb!

I can't recall right now which problem I had with your program. I thinks it couldn't find a version for the delta and kept trying to download a full version.

I followed the script, but it was easier for me to find out the process than to find the error (either yours or mine)

Now I'm emerging version 3.3 (I was using version 3.2) and I'll give it another try.

I'll tell you about in some other thread

Saludos

Sas

PS: Thanks to you and the getdelta guys I can keep my gentoo box upto date

PS2: Forgive me for this off topic intrusion

Yar-Tour · Post by **Yar-Tour** » Wed Aug 03, 2005 7:07 am

Hello ferringb, thanks you for great work, but i'm cannot enter the linux01.gwdg.de/~nlissne/ - it is "Forbidden" What's wrong?

opensas · Post by **opensas** » Wed Aug 03, 2005 3:17 pm

Hi all!

There's a bunch of people (including me, of course) that are experiencign troubles trying to access deltup server. (just have a look at the following thread:

We're getting the following error:

Code: Select all


>>> Downloading http://distfiles.gentoo.org/distfiles/kdelibs-3.4.2.tar.bz2
Searching for a previously downloaded file in /mnt/reiser/usr/portage//distfiles

We have following candidates to choose from
kdelibs-3.3.2.tar.bz2
kdelibs-3.4.0.tar.bz2
kdelibs-3.4.1.tar.bz2

The best of all is ... kdelibs-3.4.1.tar.bz2

Checking if this file is OK.

Trying to download kdelibs-3.4.1.tar.bz2-kdelibs-3.4.2.tar.bz2.dtu

--08:44:23--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=kdelibs-3.4.1.tar.bz2&want=kdelibs-3.4.2.tar.bz2&
url=ftp://download.au.kde.org/pub/kde/stable/3.4.2/src/kdelibs-3.4.2.tar.bz2&version=0.7&time=1122626663
           => `deltup.php?have=kdelibs-3.4.1.tar.bz2&want=kdelibs-3.4.2.tar.bz2&url=ftp:%2F%2Fdownload.au.kde.org%2F
pub%2Fkde%2Fstable%2F3.4.2%2Fsrc%2Fkdelibs-3.4.2.tar.bz2&version=0.7&time=1122626663'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
08:44:34 ERROR 403: Forbidden.

The dtu could not be fetched, downloading full file from original URL

--08:44:34--  http://distfiles.gentoo.org/distfiles/kdelibs-3.4.2.tar.bz2
           => `kdelibs-3.4.2.tar.bz2'
Resolving distfiles.gentoo.org... 156.56.247.195, 140.211.166.134
Connecting to distfiles.gentoo.org[156.56.247.195]:80...

So if anybody can have a look at it (blackpenguin or ferringb, both of which have always benn willing to help us with these stuffs

)
we'll be terribly thankful

Thanks

Saludos

Sas

Saludos

Stephonovich · Post by **Stephonovich** » Fri Aug 26, 2005 1:27 pm

Anyone else had a problem with gtk+-2.8.2? I've been trying to fetch it for the past couple of days (from gtk+-2.8.0), and I just get caught in the endless (almost - 15 minutes according to the .rc file) loop of "Server will create the dtu file now".

I've only had it create a file for me one time before, and it worked fine then; took maybe 5 attempts before it was ready. I'm assuming it just has a counter of the number of times people try to access a certain file, and after that time, it creates the dtu; one person being the lucky (or unlucky) one who gets to watch it.

dusanc · Post by **dusanc** » Tue Sep 20, 2005 8:15 pm

Great program.
1.Is there any way to deltup to know when we give emerge -f flag so it will first fetch all the diffs and then apply them all togeather. I'm asking because I'm on a dial-up and slow machine, so applying diff takes long and in those pauses I'm only wasting internet time.

2. Can there be diff's from eg. stage3 2005.0 and 2005.1 instalation cdroms so that people who installed gentoo from stage 3 could use this program to emerge -uD world. Currently I cannot do that because most 2005.0 packages are too old, so server doesn't have them, and cannot make diffs.

TIA
Dushan

makomk · Post by **makomk** » Sun Oct 16, 2005 1:18 pm

deltup is very nice; it's just a shame it can't generate useful deltas for openoffice-bin (it's a tarball of RPMs, and therefore not amenable to delta generation - most of the data within them is compressed), especially as it's one of the larger packages.

Veneroso · Post by **Veneroso** » Sun Oct 16, 2005 9:16 pm

I'm not sure what's going on here, but the deltup server says that the dtu exists, but when it redirects to the mirror, i get a 404 error.

Code: Select all

Calculating world dependencies ...done!
>>> emerge (1 of 3) dev-lang/python-2.4.2 to /
>>> Downloading http://csociety-ftp.ecn.purdue.edu/pub/gentoo/distfiles/Python-2.4.2.tar.bz2
Searching for a previously downloaded file in /usr/portage/distfiles

We have following candidates to choose from
Python-2.3.4.tar.bz2
Python-2.3.5.tar.bz2
Python-2.4.1.tar.bz2

The best of all is ... Python-2.4.1.tar.bz2

Checking if this file is OK.

----------------

Edit:  It seemed to work fine for the other 2 upgrade packages today.  I also modified my mirror list in make.conf, but I am not sure if this was the solution or not.

Trying to download Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu

--15:06:43--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&
url=http://www.python.org/ftp/python/2.4.2/Python-2.4.2.tar.bz2&version=0.7&time=1129489603
           => `deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&url=http:%2F%2Fwww.python.org%2Fftp%2F
python%2F2.4.2%2FPython-2.4.2.tar.bz2&version=0.7&time=1129489603'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu [following]
--15:07:15--  http://134.76.13.21/%7Enlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu
           => `Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 404 Not Found
15:07:30 ERROR 404: Not Found.

The dtu could not be fetched, downloading full file from original URL

Is anyone else experiencing this?

I see it with most of the packages I try to upgrade.

Yar-Tour · Post by **Yar-Tour** » Thu Oct 20, 2005 9:09 pm

Veneroso wrote:I'm not sure what's going on here, but the deltup server says that the dtu exists, but when it redirects to the mirror, i get a 404 error.

Code: Select all

Calculating world dependencies ...done!
>>> emerge (1 of 3) dev-lang/python-2.4.2 to /
>>> Downloading http://csociety-ftp.ecn.purdue.edu/pub/gentoo/distfiles/Python-2.4.2.tar.bz2
Searching for a previously downloaded file in /usr/portage/distfiles

We have following candidates to choose from
Python-2.3.4.tar.bz2
Python-2.3.5.tar.bz2
Python-2.4.1.tar.bz2

The best of all is ... Python-2.4.1.tar.bz2

Checking if this file is OK.

----------------

Edit:  It seemed to work fine for the other 2 upgrade packages today.  I also modified my mirror list in make.conf, but I am not sure if this was the solution or not.

Trying to download Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu

--15:06:43--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&
url=http://www.python.org/ftp/python/2.4.2/Python-2.4.2.tar.bz2&version=0.7&time=1129489603
           => `deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&url=http:%2F%2Fwww.python.org%2Fftp%2F
python%2F2.4.2%2FPython-2.4.2.tar.bz2&version=0.7&time=1129489603'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu [following]
--15:07:15--  http://134.76.13.21/%7Enlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu
           => `Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 404 Not Found
15:07:30 ERROR 404: Not Found.

The dtu could not be fetched, downloading full file from original URL

Is anyone else experiencing this?

I see it with most of the packages I try to upgrade.

Yes, i'am also suffer from this, but IMHO this error raised when dtu file is on main server linux01.gwdg.de. FTP works.

steve_v · Post by **steve_v** » Fri Oct 28, 2005 9:45 am

It looks like the server is directing us to deltas that no longer exist (for some reason).
Bugger.

steve_v · Post by **steve_v** » Sat Oct 29, 2005 6:07 am

Any ideas on these 404 errors?

Code: Select all

--18:54:43--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=audacity-src-1.2.1.tar.bz2&want=audacity-src-1.2.3.tar.gz&url=&version=0.7&time=1130565283
           => `deltup.php?have=audacity-src-1.2.1.tar.bz2&want=audacity-src-1.2.3.tar.gz&url=&version=0.7&time=1130565283'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltas/audacity-src-1.2.1.tar.bz2-audacity-src-1.2.3.tar.gz.dtu [following]
--18:54:46--  http://134.76.13.21/%7Enlissne/deltas/audacity-src-1.2.1.tar.bz2-audacity-src-1.2.3.tar.gz.dtu
           => `audacity-src-1.2.1.tar.bz2-audacity-src-1.2.3.tar.gz.dtu'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 404 Not Found
18:54:47 ERROR 404: Not Found.

Hmm??

Yar-Tour · Post by **Yar-Tour** » Mon Oct 31, 2005 9:39 pm

I send an e-mail with description of "404-error" on nlissne@linux01.gwdg.de, but didn't recieve an answer

jbergler · Post by **jbergler** » Sun Nov 20, 2005 10:00 am

switching to bdelta seems to require the bpacth util but the ebuild doesnt depend on it.

genstef · Post by **genstef** » Sun Nov 20, 2005 12:30 pm

jbergler wrote:switching to bdelta seems to require the bpacth util but the ebuild doesnt depend on it.

It is included in the ebuild:

Code: Select all

# qlist bdelta
/usr/bin/bdelta
/usr/bin/bpatch
/usr/lib/libbdelta.so
/usr/share/doc/bdelta-0.1.0/README.gz

steve_v · Post by **steve_v** » Mon Nov 21, 2005 5:12 am

Looks I'm connecting to a different mirror now, 404 errors gone

How to reduce your download traffic by 75% or more

dynamic deltup servers work with amd64 now, too

Feature reques and some troubles

404s

bumpity bump

404