Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to reduce your download traffic by 75% or more
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... , 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Wed May 11, 2005 10:37 pm    Post subject: dynamic deltup servers work with amd64 now, too Reply with quote

Announcement: getdelta.sh and the dynamic deltup server network can be used with amd64 now, too

This is possible, because we have switched from xdelta to bdelta - and bdelta does not only create smaller patches, it also is perfectly amd64 compatible.

Welcome to all amd64-users who like to use deltup for updating.

Have fun.
blackpenguin
Back to top
View user's profile Send private message
localghost
Apprentice
Apprentice


Joined: 28 Oct 2004
Posts: 185
Location: Sint-Niklaas, Belgium

PostPosted: Thu May 12, 2005 10:49 am    Post subject: Reply with quote

prymitive wrote:
thomasa88 wrote:
Sweet prg :D

(kernel-6.8.2 -> 6.8.11-r8 = 5 MB instead of 36 :D)
(I had kernel 6.8.11-r6 b4 :P)


where did You get that stuff?? Did Linus send You a beta of 6.8.11 kernel that he is developing in secret? (or did You ment 2.6.8 -> 2.6.11 but numers are not Your domain ;) )??


He's probably dyscalculic. :oops:
Don't attribute to malice what can easily be attributed to stupidity. (Murphy, if I'm correct)
_________________
411 /0µr 84$3 4r3 8310n9 70 µ$.
Back to top
View user's profile Send private message
opensas
Guru
Guru


Joined: 24 Nov 2004
Posts: 408
Location: Buenos Aires - Argentina

PostPosted: Sat Jun 04, 2005 6:16 am    Post subject: Reply with quote

Hi blackpenguin, just a suggestion to add to getdelta.rc


When dealing with small files (something between 20k ~ 100k) it's more efficient to just download thefile than going to deltup server, build the diff, donloading, unpacking, patching, etc...

It would be nice to have a setting in getdelta.rc to establish the minimum size of the file to use deltup, if the required file is greater, then just donwload the whole thing. (Something like MINIMUN_TARGET_SIZE)

Optionally, to complete avoid going to the deltup server, there acould be another setting for skiping the file if the previously downloaded file exceeds a certain size, that is we would be assuming that the required file has more or less the same size than the previous one. (Something like MINIMUN_SOURCE_SIZE)

Well, it's just an idea...

Saludos

Sas

PS: Right know I'm upgrading my whole kde 3.4 to 3.4.1... and it's working like a charm. :wink:

(Obviously, lots of people have done the same before me, that's what I found every diff file I need)

Without getdelta, I wouldn't have even thought about downloading the whole kde stuff thru the phone line.

Once again, great work blackpenguin :lol:

BTW, I just emerged emerge-delta-webrsync. I decided to give it a try. (are you involved in that project too???)
Back to top
View user's profile Send private message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Wed Jun 15, 2005 2:36 pm    Post subject: Reply with quote

I just emerged getdelta, but can't seem to find /etc/deltup/getdelta.rc ???
In fact even the /etc/deltup folder doesnt exist
Something wrong with the ebuild?
_________________
eschew obfuscation
Back to top
View user's profile Send private message
Pseud
Apprentice
Apprentice


Joined: 19 Mar 2004
Posts: 273
Location: Bangalore, India

PostPosted: Wed Jun 15, 2005 4:14 pm    Post subject: Reply with quote

Oh, heh.
getdelta.rc was created the very next time I tried using emerge :oops:
_________________
eschew obfuscation
Back to top
View user's profile Send private message
ttuttle
Tux's lil' helper
Tux's lil' helper


Joined: 03 Oct 2004
Posts: 131

PostPosted: Sat Jun 18, 2005 11:39 pm    Post subject: Reply with quote

Would this be useful for broadband users as well? I have DSL, but some packages (OpenOffice, X.org, and so on) are still quite big.

Perhaps, if you added the "minimum file size" option, you could also add an option to choose a good value for it based on connection speed and processor speed. (The idea being that the decision is a balance between using bandwidth to download extra data and using processor time to unpack/repack just the changes.)
Back to top
View user's profile Send private message
localghost
Apprentice
Apprentice


Joined: 28 Oct 2004
Posts: 185
Location: Sint-Niklaas, Belgium

PostPosted: Mon Jun 20, 2005 10:13 am    Post subject: Reply with quote

ThinkingInBinary wrote:
Would this be useful for broadband users as well? I have DSL, but some packages (OpenOffice, X.org, and so on) are still quite big.

Perhaps, if you added the "minimum file size" option, you could also add an option to choose a good value for it based on connection speed and processor speed. (The idea being that the decision is a balance between using bandwidth to download extra data and using processor time to unpack/repack just the changes.)


I have cable (4.4 Mbps down, 192 Kbps up), and I use it.
I don't care how long it takes, my compile farm is fassssst enough. :wink:
_________________
411 /0µr 84$3 4r3 8310n9 70 µ$.
Back to top
View user's profile Send private message
larryOnOff
n00b
n00b


Joined: 10 Feb 2005
Posts: 38

PostPosted: Tue Jul 19, 2005 6:25 am    Post subject: Reply with quote

Can getdelta download portage-*.tar.bz2?
It will be helpfull, very helpful.
Back to top
View user's profile Send private message
lefou
Apprentice
Apprentice


Joined: 18 Feb 2004
Posts: 199
Location: Germany, Lusatia

PostPosted: Tue Jul 19, 2005 6:51 am    Post subject: Reply with quote

You should try app-portage/emerge-delta-webrsync. This is what you want. :)

lefou
Back to top
View user's profile Send private message
opensas
Guru
Guru


Joined: 24 Nov 2004
Posts: 408
Location: Buenos Aires - Argentina

PostPosted: Tue Jul 26, 2005 4:46 am    Post subject: Reply with quote

I do recommend to at least give a try to emerge-delta-webrsync

The script was a bit buggy, but the whole process is quite simple

I'll give you a step by step example:

first emerge tarsync (so that you can sync the portage tree without having to untar the file

then you need a tar file, like portage-20050722.tar

you can one bzipped tar file get it from http://gentoo.osuosl.org/snapshots/

all you have to do is download it and unzip it

then you donwload the following the delta files files:

Code:

wget -nc http://egret.gentoo.org/~gpatches/snapshot-deltas/snapshot-20050722-20050723.patch.bz2
wget -nc http://egret.gentoo.org/~gpatches/snapshot-deltas/snapshot-20050723-20050724.patch.bz2


and the unziped md5sum

Code:

wget -nc http://egret.gentoo.org/~gpatches/snapshots/portage-20050724.tar.bz2.umd5sum


and run the following

Code:

patcher portage-20050722.tar snapshot-20050722-20050723.patch.bz2 snapshot-20050723-20050724.patch.bz2 portage-20050724.tar 


we check the new file

Code:

md5sum -c portage-20050724.tar.bz2.umd5sum


we should get something like

Code:

portage-20050724.tar: OK


if everything works OK, then we can sync our portage tree from the tar file with the following command

Code:

tarsync portage-20050724.tar /usr/portage -v -s 1 -o portage -g portage -e /distfiles -e /packages -e /local


then we issue an

Code:

emerge metadata


and in my case I also issue an

Code:

update-eix


With all this in mind it shouldn't be difficult (for someone with some minimun bash knowledge, wich is unfortunately not my case) to make a script to automatize these tasks

Saludos

Sas

PS: Take into account that every delta file is about 100KB, and it's quite unusual to have some delta bigger than 200KB, and each bzipped tar file is 19MB, so you can get an idea of the download time saving!!!
Back to top
View user's profile Send private message
ferringb
Retired Dev
Retired Dev


Joined: 03 Apr 2003
Posts: 357

PostPosted: Tue Jul 26, 2005 4:51 am    Post subject: Reply with quote

opensas wrote:
I do recommend to at least give a try to emerge-delta-webrsync
The script was a bit buggy, but the whole process is quite simple

Buggy how?


opensas wrote:
With all this in mind it shouldn't be difficult (for someone with some minimun bash knowledge, wich is unfortunately not my case) to make a script to automatize these tasks

This is what emerge-delta-webrsync does already (and it's in bash) :)

opensas wrote:
PS: Take into account that every delta file is about 100KB, and it's quite unusual to have some delta bigger than 200KB, and each bzipped tar file is 19MB, so you can get an idea of the download time saving!!!

Occasionally peaks to 300kB, but yeah, savings are on the order of ~99%
Back to top
View user's profile Send private message
opensas
Guru
Guru


Joined: 24 Nov 2004
Posts: 408
Location: Buenos Aires - Argentina

PostPosted: Tue Jul 26, 2005 7:46 am    Post subject: Reply with quote

Hi ferringb!

I can't recall right now which problem I had with your program. I thinks it couldn't find a version for the delta and kept trying to download a full version.

I followed the script, but it was easier for me to find out the process than to find the error (either yours or mine)

Now I'm emerging version 3.3 (I was using version 3.2) and I'll give it another try.

I'll tell you about in some other thread

Saludos

Sas

PS: Thanks to you and the getdelta guys I can keep my gentoo box upto date :lol: :lol: :lol:

PS2: Forgive me for this off topic intrusion
Back to top
View user's profile Send private message
Yar-Tour
n00b
n00b


Joined: 03 Aug 2005
Posts: 8

PostPosted: Wed Aug 03, 2005 7:07 am    Post subject: Reply with quote

Hello ferringb, thanks you for great work, but i'm cannot enter the linux01.gwdg.de/~nlissne/ - it is "Forbidden" What's wrong?
Back to top
View user's profile Send private message
opensas
Guru
Guru


Joined: 24 Nov 2004
Posts: 408
Location: Buenos Aires - Argentina

PostPosted: Wed Aug 03, 2005 3:17 pm    Post subject: Reply with quote

Hi all!


There's a bunch of people (including me, of course) that are experiencign troubles trying to access deltup server. (just have a look at the following thread:


We're getting the following error:

Code:


>>> Downloading http://distfiles.gentoo.org/distfiles/kdelibs-3.4.2.tar.bz2
Searching for a previously downloaded file in /mnt/reiser/usr/portage//distfiles

We have following candidates to choose from
kdelibs-3.3.2.tar.bz2
kdelibs-3.4.0.tar.bz2
kdelibs-3.4.1.tar.bz2

The best of all is ... kdelibs-3.4.1.tar.bz2

Checking if this file is OK.

Trying to download kdelibs-3.4.1.tar.bz2-kdelibs-3.4.2.tar.bz2.dtu

--08:44:23--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=kdelibs-3.4.1.tar.bz2&want=kdelibs-3.4.2.tar.bz2&
url=ftp://download.au.kde.org/pub/kde/stable/3.4.2/src/kdelibs-3.4.2.tar.bz2&version=0.7&time=1122626663
           => `deltup.php?have=kdelibs-3.4.1.tar.bz2&want=kdelibs-3.4.2.tar.bz2&url=ftp:%2F%2Fdownload.au.kde.org%2F
pub%2Fkde%2Fstable%2F3.4.2%2Fsrc%2Fkdelibs-3.4.2.tar.bz2&version=0.7&time=1122626663'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
08:44:34 ERROR 403: Forbidden.

The dtu could not be fetched, downloading full file from original URL

--08:44:34--  http://distfiles.gentoo.org/distfiles/kdelibs-3.4.2.tar.bz2
           => `kdelibs-3.4.2.tar.bz2'
Resolving distfiles.gentoo.org... 156.56.247.195, 140.211.166.134
Connecting to distfiles.gentoo.org[156.56.247.195]:80...



So if anybody can have a look at it (blackpenguin or ferringb, both of which have always benn willing to help us with these stuffs :wink: )
we'll be terribly thankful

Thanks

Saludos

Sas

Saludos
Back to top
View user's profile Send private message
Stephonovich
n00b
n00b


Joined: 20 May 2004
Posts: 72
Location: United States

PostPosted: Fri Aug 26, 2005 1:27 pm    Post subject: Reply with quote

Anyone else had a problem with gtk+-2.8.2? I've been trying to fetch it for the past couple of days (from gtk+-2.8.0), and I just get caught in the endless (almost - 15 minutes according to the .rc file) loop of "Server will create the dtu file now".

I've only had it create a file for me one time before, and it worked fine then; took maybe 5 attempts before it was ready. I'm assuming it just has a counter of the number of times people try to access a certain file, and after that time, it creates the dtu; one person being the lucky (or unlucky) one who gets to watch it.
_________________
Who needs reincarnation when you've got parallel universes?
Back to top
View user's profile Send private message
dusanc
Apprentice
Apprentice


Joined: 19 Sep 2005
Posts: 248
Location: Serbia

PostPosted: Tue Sep 20, 2005 8:15 pm    Post subject: Feature reques and some troubles Reply with quote

Great program.
1.Is there any way to deltup to know when we give emerge -f flag so it will first fetch all the diffs and then apply them all togeather. I'm asking because I'm on a dial-up and slow machine, so applying diff takes long and in those pauses I'm only wasting internet time.

2. Can there be diff's from eg. stage3 2005.0 and 2005.1 instalation cdroms so that people who installed gentoo from stage 3 could use this program to emerge -uD world. Currently I cannot do that because most 2005.0 packages are too old, so server doesn't have them, and cannot make diffs.

TIA
Dushan
Back to top
View user's profile Send private message
makomk
n00b
n00b


Joined: 15 Jul 2005
Posts: 46
Location: Not all there

PostPosted: Sun Oct 16, 2005 1:18 pm    Post subject: Reply with quote

deltup is very nice; it's just a shame it can't generate useful deltas for openoffice-bin (it's a tarball of RPMs, and therefore not amenable to delta generation - most of the data within them is compressed), especially as it's one of the larger packages.
Back to top
View user's profile Send private message
Veneroso
n00b
n00b


Joined: 26 May 2004
Posts: 27
Location: Batavia, New York

PostPosted: Sun Oct 16, 2005 9:16 pm    Post subject: Reply with quote

I'm not sure what's going on here, but the deltup server says that the dtu exists, but when it redirects to the mirror, i get a 404 error.

Code:

Calculating world dependencies ...done!
>>> emerge (1 of 3) dev-lang/python-2.4.2 to /
>>> Downloading http://csociety-ftp.ecn.purdue.edu/pub/gentoo/distfiles/Python-2.4.2.tar.bz2
Searching for a previously downloaded file in /usr/portage/distfiles

We have following candidates to choose from
Python-2.3.4.tar.bz2
Python-2.3.5.tar.bz2
Python-2.4.1.tar.bz2

The best of all is ... Python-2.4.1.tar.bz2

Checking if this file is OK.

----------------

Edit:  It seemed to work fine for the other 2 upgrade packages today.  I also modified my mirror list in make.conf, but I am not sure if this was the solution or not.

Trying to download Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu

--15:06:43--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&
url=http://www.python.org/ftp/python/2.4.2/Python-2.4.2.tar.bz2&version=0.7&time=1129489603
           => `deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&url=http:%2F%2Fwww.python.org%2Fftp%2F
python%2F2.4.2%2FPython-2.4.2.tar.bz2&version=0.7&time=1129489603'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu [following]
--15:07:15--  http://134.76.13.21/%7Enlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu
           => `Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 404 Not Found
15:07:30 ERROR 404: Not Found.

The dtu could not be fetched, downloading full file from original URL


Is anyone else experiencing this?

I see it with most of the packages I try to upgrade.
Back to top
View user's profile Send private message
Yar-Tour
n00b
n00b


Joined: 03 Aug 2005
Posts: 8

PostPosted: Thu Oct 20, 2005 9:09 pm    Post subject: Reply with quote

Veneroso wrote:
I'm not sure what's going on here, but the deltup server says that the dtu exists, but when it redirects to the mirror, i get a 404 error.

Code:

Calculating world dependencies ...done!
>>> emerge (1 of 3) dev-lang/python-2.4.2 to /
>>> Downloading http://csociety-ftp.ecn.purdue.edu/pub/gentoo/distfiles/Python-2.4.2.tar.bz2
Searching for a previously downloaded file in /usr/portage/distfiles

We have following candidates to choose from
Python-2.3.4.tar.bz2
Python-2.3.5.tar.bz2
Python-2.4.1.tar.bz2

The best of all is ... Python-2.4.1.tar.bz2

Checking if this file is OK.

----------------

Edit:  It seemed to work fine for the other 2 upgrade packages today.  I also modified my mirror list in make.conf, but I am not sure if this was the solution or not.

Trying to download Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu

--15:06:43--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&
url=http://www.python.org/ftp/python/2.4.2/Python-2.4.2.tar.bz2&version=0.7&time=1129489603
           => `deltup.php?have=Python-2.4.1.tar.bz2&want=Python-2.4.2.tar.bz2&url=http:%2F%2Fwww.python.org%2Fftp%2F
python%2F2.4.2%2FPython-2.4.2.tar.bz2&version=0.7&time=1129489603'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu [following]
--15:07:15--  http://134.76.13.21/%7Enlissne/deltas/Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu
           => `Python-2.4.1.tar.bz2-Python-2.4.2.tar.bz2.dtu'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 404 Not Found
15:07:30 ERROR 404: Not Found.

The dtu could not be fetched, downloading full file from original URL


Is anyone else experiencing this?

I see it with most of the packages I try to upgrade.


Yes, i'am also suffer from this, but IMHO this error raised when dtu file is on main server linux01.gwdg.de. FTP works.
Back to top
View user's profile Send private message
steve_v
Guru
Guru


Joined: 20 Jun 2004
Posts: 386
Location: New Zealand

PostPosted: Fri Oct 28, 2005 9:45 am    Post subject: 404s Reply with quote

It looks like the server is directing us to deltas that no longer exist (for some reason).
Bugger.
Back to top
View user's profile Send private message
steve_v
Guru
Guru


Joined: 20 Jun 2004
Posts: 386
Location: New Zealand

PostPosted: Sat Oct 29, 2005 6:07 am    Post subject: bumpity bump Reply with quote

Any ideas on these 404 errors?

Code:

--18:54:43--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=audacity-src-1.2.1.tar.bz2&want=audacity-src-1.2.3.tar.gz&url=&version=0.7&time=1130565283
           => `deltup.php?have=audacity-src-1.2.1.tar.bz2&want=audacity-src-1.2.3.tar.gz&url=&version=0.7&time=1130565283'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://134.76.13.21/~nlissne/deltas/audacity-src-1.2.1.tar.bz2-audacity-src-1.2.3.tar.gz.dtu [following]
--18:54:46--  http://134.76.13.21/%7Enlissne/deltas/audacity-src-1.2.1.tar.bz2-audacity-src-1.2.3.tar.gz.dtu
           => `audacity-src-1.2.1.tar.bz2-audacity-src-1.2.3.tar.gz.dtu'
Connecting to 134.76.13.21:80... connected.
HTTP request sent, awaiting response... 404 Not Found
18:54:47 ERROR 404: Not Found.


Hmm??
Back to top
View user's profile Send private message
Yar-Tour
n00b
n00b


Joined: 03 Aug 2005
Posts: 8

PostPosted: Mon Oct 31, 2005 9:39 pm    Post subject: Reply with quote

I send an e-mail with description of "404-error" on nlissne@linux01.gwdg.de, but didn't recieve an answer :(
Back to top
View user's profile Send private message
jbergler
n00b
n00b


Joined: 20 Nov 2005
Posts: 2

PostPosted: Sun Nov 20, 2005 10:00 am    Post subject: Reply with quote

switching to bdelta seems to require the bpacth util but the ebuild doesnt depend on it.
Back to top
View user's profile Send private message
genstef
Retired Dev
Retired Dev


Joined: 13 Jun 2004
Posts: 668
Location: M/Bay/Germany

PostPosted: Sun Nov 20, 2005 12:30 pm    Post subject: Reply with quote

jbergler wrote:
switching to bdelta seems to require the bpacth util but the ebuild doesnt depend on it.

It is included in the ebuild:
Code:
# qlist bdelta
/usr/bin/bdelta
/usr/bin/bpatch
/usr/lib/libbdelta.so
/usr/share/doc/bdelta-0.1.0/README.gz
Back to top
View user's profile Send private message
steve_v
Guru
Guru


Joined: 20 Jun 2004
Posts: 386
Location: New Zealand

PostPosted: Mon Nov 21, 2005 5:12 am    Post subject: 404 Reply with quote

Looks I'm connecting to a different mirror now, 404 errors gone :-)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3 ... , 9, 10, 11  Next
Page 10 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum