How to reduce your download traffic by 75% or more

[TPC]

hm.. it doesn't do firefox-0.9.3-source.tar.bz2 to firefox-1.0PR-source.tar.bz2

thats 30mb of downloading when there could be less than 1

[MooktaKiNG]

blackpenguin:
If you set the script to also ignore the occurences of "PR" the firefox download works. However, i think the deltup server is not configured to do this.
Please could you fix this. I just want to see how much it saves from a 0.9.3 version of firefox Thanks

[Greven]

*BUMP*

It's working great but I don't want to download FireFoxPR fully due to problem.....


Thanks....
Jason

[blackpenguin]

The new scheme to detect thirdpartymirrors failed to detect the correct path on the mirrors (well, it's beta-software)

Fixed in version 0.5.3.4 of getdelta.sh

[PrakashP]

Are you sure you uploaded the fixed script? It still looks like 0.5.3.3 to me...

[blackpenguin]

Are you sure you uploaded the fixed script? It still looks like 0.5.3.3 to me...

Yes, and I've double checked this. Have a look at line 29 which should say:

# version 0.5.3.4 2004/09/20
# - fixed wrong path-detection with thirdpartymirrors

[PrakashP]

Ah OK, my fault, sorry.

[ThomasL]

Nice script.

But I don't like this update:

Code: Select all

have=linux-2.6.8.1.tar.bz2
want=links-2.1pre15.tar.bz2
version=0.5.3.4

The script tries to update links and takes the linux-kernel as base for xdelta. I'm sure this diff will be much bigger than links-2.1pre15.tar.bz2.

Any ideas how to remove this problem?

Many Thanks.

[kaktyc]

Thanks a lot! It's a great thing! Especially for modem users (like me )

I have a slow CPU (Cel 375) and applying patch takes a long time. For example, I downloaded dtu for gcc-3.3.4 (~750 kb). Deltup applied patch for about 5 mins.
I think, it will be usefull to add an option to download only dtu files and after manual disconnect apply patches.
But portage checks every package, so you can download files form -pf output.

Sorry for English

[LucaSpiller]

Wow, this is brilliant! Fantastic!

I decided to try it out by updating Xorg (eeek). The package is about 70mb - or something like that - and would have taken me hours to get. With this it took just a few minutes. Now have you got any ideas of how to speed up compiling?

Also, in a rather strange problem (slightly off topic) - over the last few days whenever I have resumed downloads the speeds have dropped to < 500bytes/s. I have tried using different mirrors, but I get the same problem all the time.... strange.

EDIT: I think my slow download speeds were due to my machine not liking compiling and downloading.

EDIT #2: Updating from qt-x11-free-3.3.2 to qt-x11-free-3.3.3 causes a seg fault.

Code: Select all

>>> emerge (1 of 31) x11-libs/qt-3.3.3 to /
>>> Downloading http://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/distfiles/qt-x11-free-3.3.3.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from
qt-x11-free-3.3.2.tar.bz2

The best of all is ... qt-x11-free-3.3.2.tar.bz2

Checking if this file is OK.

Trying to download qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu

--15:40:52--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=qt-x11-free-3.3.2.tar.bz2&want=qt-x11-free-3.3.3.tar.bz2&version=0.5.3
           => `deltup.php?have=qt-x11-free-3.3.2.tar.bz2&want=qt-x11-free-3.3.3.tar.bz2&version=0.5.3'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: ftp://134.102.120.44/deltup/qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu [following]
--15:40:53--  ftp://134.102.120.44/deltup/qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu
           => `qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu.3'
Connecting to 134.102.120.44:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /deltup ... done.
==> PASV ... done.    ==> RETR qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu ... done.

    [                <=>                  ] 854,616        3.11K/s

15:44:18 (4.13 KB/s) - `qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu.3' saved [854616]

GOT qt-x11-free-3.3.2.tar.bz2-qt-x11-free-3.3.3.tar.bz2.dtu.3

Yes, we got the dtu-file now let's build qt-x11-free-3.3.3.tar.bz2...

/usr/local/bin/getdelta.sh: line 536: 29774 Segmentation fault      deltup -p -v ${best_candidate}-${NEW_FILE}.dtu
The dtu could not be fetched, downloading full file from original URL

[multani]

Hi !

I use deltup since about two weeks now, without any problem. Today, I got this strange things :

Code: Select all

linathlon jon # emerge gdb
Calculating dependencies ...done!
>>> emerge (1 of 1) sys-devel/gdb-6.0 to /
>>> Downloading ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/distfiles/gdb-6.0.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from 
gdm-2.4.4.7.tar.bz2
gdm-2.6.0.3.tar.bz2
gdm-2.6.0.4.tar.bz2 

The best of all is ... gdm-2.6.0.4.tar.bz2

Checking if this file is OK.

Could not find a digest-file for gdm-2.6.0.4.tar.bz2. Testing file integrity with tar.

Trying to download gdm-2.6.0.4.tar.bz2-gdb-6.0.tar.bz2.dtu

[...]

Finally, i got my gdb debugger, but is it normal that it's dealling something with gdm ??

[ABVGD]

At first, Thank You very much, blackpenguin, for you job!

maybe it would be a good idea to check the gcc version, and patch file.h if going to use gcc 3.4.x so that it compiles.. having to fix that manually its a pain in the XXX

Code: Select all

src_unpack () {                                                                                                                                                       
        unpack ${A}                                                                                                                                                   
        cd ${S}                                                                                                                                                                             
        GCC_MAJOR_VER=$(gcc-config -c | awk -F- '{print $NF}' | cut -d"." -f1)                                                                                                               
        GCC_MINOR_VER=$(gcc-config -c | awk -F- '{print $NF}' | cut -d"." -f2)
        if [[ "$GCC_MAJOR_VER" -eq "3" && "$GCC_MINOR_VER" -ge "4" ]]; then
            epatch ${FILESDIR}/${P}-gcc34.patch                                                                                                                                   
        fi
}

Just add it to ebuild before "src_install ()" section and put "inherit eutils" to the first uncommented string in the ebuild.
In this case the patch has name ${PORTDIR_OVERLAY}/app-portage/deltup/files/deltup-0.4.0-gcc34.patch

[F1uX]

To me too the deltup executable gives a segmentation fault:

Code: Select all

GOT gcc-3.4.1.tar.bz2-gcc-3.4.2.tar.bz2.dtu.2

Yes, we got the dtu-file now let's build gcc-3.4.2.tar.bz2...

/usr/local/bin/getdelta.sh: line 536:  2918 Segmentation fault      deltup -p -v ${best_candidate}-${NEW_FILE}.dtu
The dtu could not be fetched, downloading full file from original URL

[scristian]

very rare I have segfault too, and I am using all the time deltup.

Thanks guys for this program, is working great and I hope will be in portage soon.

[Giuliani]

hi,
how i can copy and paste my output messages ?
After execute emerge system for second time, but now with deltup, i'm received some error messages like "command not found" or "to many arguments type mkdir --help....".

Sorry for my english

i'm using last deltup version (2004/09/20)

Thank's

[grobister]

wow... .can't wait to get home tonight and try this !

[eNTi]

damnit! i want that brilliant thing in the official tree!

update: i've filed a request.

[ABVGD]

When I tried to download openoffice-bin-1.1.3 (77 Mb) I've got the following:

Code: Select all

---skiped---
The best of all is ... OOo_1.1.2_LinuxIntel_install.tar.gz

Checking if this file is OK.

Trying to download 
---skiped---
OOo_1.1.2_LinuxIntel_install.tar.gz-OOo_1.1.3_LinuxIntel_install.tar.gz.dtu
Connecting to 134.102.120.44:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /deltup ... done.
==> PASV ... done.    ==> RETR OOo_1.1.2_LinuxIntel_install.tar.gz-OOo_1.1.3_LinuxIntel_install.tar.gz.dtu ... done.

    [                                                                                                            <=>             ] 30,435,680     8.01K/s             

01:36:33 (7.99 KB/s) - Control connection closed.
Giving up.

The dtu could not be fetched, downloading full file from original URL
^C

A: How to resolve this situation and to get not the full sources tarball in such case?
Q:
- Resume downloading by your preffered method. Ex.:

Code: Select all

# cd /usr/portage/distfiles
# ncftpget ftp://134.102.120.44/deltup/OOo_1.1.2_LinuxIntel_install.tar.gz-OOo_1.1.3_LinuxIntel_install.tar.gz.dtu

- Then do:

Code: Select all

# deltup -p -v OOo_1.1.2_LinuxIntel_install.tar.gz-OOo_1.1.3_LinuxIntel_install.tar.gz.dtu
OOo_1.1.2_LinuxIntel_install.tar.gz -> OOo_1.1.3_LinuxIntel_install.tar.gz: OK
cleaning up

- Checking:

Code: Select all

# md5sum  OOo_1.1.3_LinuxIntel_install.tar.gz
1d87f6c730b6fa5252d7dfd4d2c69de4  OOo_1.1.3_LinuxIntel_install.tar.gz
# cat /usr/portage/app-office/openoffice-bin/files/digest-openoffice-bin-1.1.3
MD5 1d87f6c730b6fa5252d7dfd4d2c69de4 OOo_1.1.3_LinuxIntel_install.tar.gz 80123557

MD5 sum's are equal.

[michael3]

hello

I would also like to get Openoffice 1.1.2 to 1.1.3. I get this error message:
not write temporary data. Possibly out of space
cleaning up

The harddisk isn't full and i do it with root.
Can somebody help me? I don't want to download the 230 MB again, because i havn't got a flatrate.

Edit: OK, i have fixed the Problem. there was a unfineshed dtu File in the distfiles directory. I have delete this file and now it works.

[LucaSpiller]

I have just upgraded to the latest version today, and when I try to do an "emerge -u --fetchonly world" I get the following:

Code: Select all

Calculating world dependencies ...done!
>>> emerge (1 of 8) sys-libs/glibc-2.3.4.20040808-r1 to /
>>> Previously fetched file: glibc-infopages-2.3.4.tar.bz2 MD5 ;-)
>>> Previously fetched file: glibc-manpages-2.3.4.tar.bz2 MD5 ;-)
>>> Previously fetched file: glibc-2.3.3.tar.bz2 MD5 ;-)
>>> Previously fetched file: glibc-2.3.4-branch-update-20040808.patch.bz2 MD5 ;-)
>>> md5 src_uri ;-) glibc-2.3.3.tar.bz2
>>> md5 src_uri ;-) glibc-manpages-2.3.4.tar.bz2
>>> md5 src_uri ;-) glibc-infopages-2.3.4.tar.bz2
>>> md5 src_uri ;-) glibc-2.3.4-branch-update-20040808.patch.bz2
>>> emerge (2 of 8) sys-devel/make-3.80-r1 to /
>>> Previously fetched file: make-3.80.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) make-3.80.tar.bz2
>>> emerge (3 of 8) dev-libs/glib-2.4.6 to /
>>> Previously fetched file: glib-2.4.6.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) glib-2.4.6.tar.bz2
>>> emerge (4 of 8) x11-misc/ttmkfdir-3.0.9-r2 to /
>>> Previously fetched file: ttmkfdir-3.0.9.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) ttmkfdir-3.0.9.tar.bz2
>>> emerge (5 of 8) x11-base/xorg-x11-6.8.0-r1 to /
>>> Previously fetched file: X11R6.8.0-src2.tar.gz MD5 ;-)
>>> Previously fetched file: gentoo-cursors-tad-0.3.1.tar.bz2 MD5 ;-)
>>> Previously fetched file: xorg-x11-6.8.0-patches-0.2.2.1.tar.bz2 MD5 ;-)
>>> Previously fetched file: X11R6.8.0-src5.tar.gz MD5 ;-)
>>> Previously fetched file: X11R6.8.0-src4.tar.gz MD5 ;-)
>>> Downloading http://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/distfiles/X11R6.8.0-src1.tar.gz
Checking if a Thirdparty-Mirror is used ...

/usr/local/bin/getdelta.sh: line 292: /tmp/getdelta.tmp: cannot overwrite existing file
No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from
X11R6.7.0-src1.tar.gz

The best of all is ... X11R6.7.0-src1.tar.gz

Checking if this file is OK.

Trying to download X11R6.7.0-src1.tar.gz-X11R6.8.0-src1.tar.gz.dtu

--21:03:02--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=X11R6.7.0-src1.tar.gz&want=X11R6.8.0-src1.tar.gz&version=0.5.4
           => `deltup.php?have=X11R6.7.0-src1.tar.gz&want=X11R6.8.0-src1.tar.gz&version=0.5.4'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://linux01.gwdg.de/~nlissne/deltup-server.msg [following]
--21:03:03--  http://linux01.gwdg.de/%7Enlissne/deltup-server.msg
           => `deltup-server.msg'
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 213 [text/plain]

100%[====================================>] 213           --.--K/s

21:03:04 (2.03 MB/s) - `deltup-server.msg' saved [213/213]

GOT deltup-server.msg

Yes, we got the dtu-file now let's build X11R6.8.0-src1.tar.gz...

/usr/local/bin/getdelta.sh: line 545: 27858 Segmentation fault      deltup -p -v ${best_candidate}-${NEW_FILE}.dtu
IMPORTANT MESSAGE FROM DELTUP-SERVER
A new version of getdelta.sh is available (0.5.4)

getdelta-0.5.4 fixes a security bug (possible symlink-attack)

please update.

You can get it here:

http://linux01.gwdg.de/~nlissne/getdelta-0.5.4.tar.bz2

---

After that it just hangs.

[elykyllek]

After that it just hangs.

0.6.0 is out now, going to give it a try.

[ts77]

as there seems to be no direct download of getdelta.sh ... could someone give a quick "how to use the gensync-source there?

[Twink]

Forbidden
You don't have permission to access /~nlissne/getdelta.sh on this server.

whats up?

[PrakashP]

Get the ebuild. Unfortunately the ebuild wouldn't want to dl the file, so I got the tar.bz2 link from the site and modified it for the 0.6 and downloaded per hand...

The good: Now I at least get deltas for kde packages. The client mirror detection really was broken. I hope I know get more deltas in future. Though I haven't made statistics, I can say that only in 20-30% in my case I got deltas.

[LucaSpiller]

You can get the latest version at:

http://linux01.gwdg.de/~nlissne/getdelta-0.6.tar.bz2