Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How to reduce your download traffic by 75% or more
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, ... 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Sun Aug 29, 2004 6:35 pm    Post subject: Reply with quote

tomaw wrote:
Works very nicely here too. Is the server capable of taking the load of people testing this out?


We will see :) the server is directly connected to the DFN - therefore it *can* take some load :). However, the *public* beta phase just began - so there is no experience how well it handles many users yet.

bp
Back to top
View user's profile Send private message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Sun Aug 29, 2004 6:43 pm    Post subject: Reply with quote

Thanks for your comment. I set GENTOO_MIRRORS="" and report back how it goes now. :)

Last edited by PrakashP on Sun Aug 29, 2004 8:14 pm; edited 1 time in total
Back to top
View user's profile Send private message
ruth
Retired Dev
Retired Dev


Joined: 07 Sep 2003
Posts: 640
Location: M / AN / BY / GER

PostPosted: Sun Aug 29, 2004 7:35 pm    Post subject: Reply with quote

hi,
@blackpenguin:
thank you...

works like a charm and saves me a lot of bandwidth...
thank you, again...

@devs:
please - make this an official gentoo feature...


regards
rootshell
_________________
"The compiler has tried twice to abort and cannot do so; therefore, compilation will now terminate."
-- IBM PL/I (F) error manual
Back to top
View user's profile Send private message
NoUseForAName
Tux's lil' helper
Tux's lil' helper


Joined: 22 Feb 2003
Posts: 84
Location: Sweden

PostPosted: Sun Aug 29, 2004 8:26 pm    Post subject: Reply with quote

Ok, found some more packages that didn't work.
* ruby when upgraded from ruby-1.8.1.tar.gz to ruby-1.8.2.preview2.tar.gz
* xinelib when upgraded from
xine-lib-1-rc3a.tar.gz to xine-lib-1-rc5.tar.gz
* xvid when upgraded from xvidcore-1.0.0-beta3.tar.bz2 to xvidcore-1.0.1.tar.bz2

This patch should fix all those problems:
Code:

--- getdelta.sh 2004-08-29 19:54:40.000000000 +0200
+++ /usr/local/bin/getdelta.sh 2004-08-29 22:12:40.025669552 +0200
@@ -215,7 +215,7 @@
 # and probably it will fail sometimes.
 #
 # we just ignore any occurence of
-# "pre","rc","[0-9],"_","-","." in the filenames and test
+# "preview","pre","rc","beta","[0-9],"_","-","." in the filenames and test
 # if they are the same (or VERY similar (differ only in 1 char)).
 # to reduce the files to check, we only check files
 # that have the same first 2 letters in name
@@ -240,7 +240,7 @@
     -e "s/src7/srcGGG/g" <<< $NEW_FILE)
  fi
 
- mask=$(sed -e "s/[0-9]//g" -e "s/pre//g" -e "s/rc//g" -e "s/[\._-]//g" -e "s/\+//g" -e "s/ //g" <<< "${MASK_FILENAME}")
+ mask=$(sed -e "s/[0-9]//g" -e "s/preview//g" -e "s/pre//g" -e "s/rc//g" -e "s/beta//g" -e "s/[\._-]//g" -e "s/\+//g" -e "s/ //g" <<< "${MASK_FILENAME}")
 
  let len1=$(wc -c <<< $mask)-1
 
@@ -259,7 +259,7 @@
     -e "s/src6/srcFFF/g" \
     -e "s/src7/srcGGG/g" <<< $name)
   fi
-  mask2=$(sed -e "s/[0-9]//g" -e "s/pre//g" -e "s/rc//g" -e "s/[\._-]//g" -e "s/\+//g" -e "s/ //g" <<< "${MASK_FILENAME}")
+  mask2=$(sed -e "s/[0-9]//g" -e "s/preview//g" -e "s/pre//g" -e "s/rc//g" -e "s/beta//g" -e "s/[\._-]//g" -e "s/\+//g" -e "s/ //g" <<< "${MASK_FILENAME}")
 
   # add any file, that results in the same mask or differ not more than one letter
 
@@ -267,9 +267,9 @@
   if [ $len1 -gt $len2 ]
   then
    max=${len1}
-   let min=${len2}-1
+   let min=${len2}
   else
-   let min=${len1}-1
+   let min=${len1}
    max=${len2}
   fi
   let df=${max}-${min}
Back to top
View user's profile Send private message
NoUseForAName
Tux's lil' helper
Tux's lil' helper


Joined: 22 Feb 2003
Posts: 84
Location: Sweden

PostPosted: Sun Aug 29, 2004 8:35 pm    Post subject: Reply with quote

Changing the default of fetch to add resuming of files would probably be nice too, like:
Code:
FETCH="/usr/bin/wget -c -t 1 --passive-ftp"
Back to top
View user's profile Send private message
NoUseForAName
Tux's lil' helper
Tux's lil' helper


Joined: 22 Feb 2003
Posts: 84
Location: Sweden

PostPosted: Sun Aug 29, 2004 9:30 pm    Post subject: Reply with quote

Ok, and another fix :D

The line:
Code:
${VERBOSITY} && echo -e "$\n{RED}The server could not build the dtu-file for ${NEW_FILE}${NORMAL}\n"| tee -a $LOGFILE

should be changed to:
Code:
${VERBOSITY} && echo -e "${RED}The server could not build the dtu-file for ${NEW_FILE}${NORMAL}\n"| tee -a $LOGFILE


(removing the \n so it actually gets printed in red and not with a newline...)
Back to top
View user's profile Send private message
Stefan1801
n00b
n00b


Joined: 20 Aug 2004
Posts: 50
Location: Marienmünster

PostPosted: Sun Aug 29, 2004 9:55 pm    Post subject: Reply with quote

how can i get those stats, like "...originial size=xxmb, saved= xx%" ?

thanks.
_________________
well, i'm drowning in time to a desperate beat
---
Router
Workstation
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Mon Aug 30, 2004 4:19 pm    Post subject: Reply with quote

Stefan1801 wrote:
how can i get those stats, like "...originial size=xxmb, saved= xx%" ?


Just have a look at http://linux01.gwdg.de/~nlissne/

bp
Back to top
View user's profile Send private message
pleusicles
n00b
n00b


Joined: 26 May 2004
Posts: 44

PostPosted: Mon Aug 30, 2004 5:44 pm    Post subject: Reply with quote

First of all, thanks for this util, it's wonderful! :D

I just found a problem regarding the search for previously dl'ed distfiles (yes, I know it's tricky as stated in the script...):

I tried to upgrade mplayer, which includes downloading the following files:

font-arial-iso-8859-1.tar.bz2 and
font-arial-iso-8859-2.tar.bz2

Of course, 1 and 2 are interpreted as version instead of being part of the name... The server builds the dtu, it's downloaded, but then the checksums don't match.

After a few tries, the whole emerge aborts instead of downloading the original file.

Just the end of the story:

Code:

>>> Downloading http://ftp3.mplayerhq.hu/MPlayer//releases/fonts/font-arial-iso-8859-1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

Chosen Mirror is mplayer

Searching for a previosly downloaded file in /store/distfiles

We have following candidates to choose from
font-arial-iso-8859-2.tar.bz2

The best of all is ... font-arial-iso-8859-2.tar.bz2

Trying to download font-arial-iso-8859-2.tar.bz2-font-arial-iso-8859-1.tar.bz2.dtu

--19:23:15--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=font-arial-iso-8859-2.tar.bz2&want=font-arial-iso-8859-1.tar.bz2&version=0.5.2&mirror=mplayer&path=/releases/fonts/
           => `deltup.php?have=font-arial-iso-8859-2.tar.bz2&want=font-arial-iso-8859-1.tar.bz2&version=0.5.2&mirror=mplayer&path=%2Freleases%2Ffonts%2F'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://linux01.gwdg.de/~nlissne/deltas/font-arial-iso-8859-2.tar.bz2-font-arial-iso-8859-1.tar.bz2.dtu [following]
--19:23:16--  http://linux01.gwdg.de/%7Enlissne/deltas/font-arial-iso-8859-2.tar.bz2-font-arial-iso-8859-1.tar.bz2.dtu
           => `font-arial-iso-8859-2.tar.bz2-font-arial-iso-8859-1.tar.bz2.dtu'
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 113,958 [application/x-bzip2]

100%[=============================================================>] 113,958       72.69K/s

19:23:17 (72.52 KB/s) - `font-arial-iso-8859-2.tar.bz2-font-arial-iso-8859-1.tar.bz2.dtu' saved [113958/113958]

GOT font-arial-iso-8859-2.tar.bz2-font-arial-iso-8859-1.tar.bz2.dtu

Yes, we got the dtu-file now let's build font-arial-iso-8859-1.tar.bz2...

font-arial-iso-8859-2.tar.bz2 -> font-arial-iso-8859-1.tar.bz2: OK
cleaning up
!!! Fetched file: font-arial-iso-8859-1.tar.bz2 MD5 FAILED! Removing corrupt distfile...
!!! Couldn't download font-arial-iso-8859-1.tar.bz2. Aborting.

!!! Fetch for /usr/portage/media-video/mplayer/mplayer-1.0_pre4-r7.ebuild failed, continuing...



!!! Some fetch errors were encountered.  Please see above for details.
Back to top
View user's profile Send private message
GurliGebis
Developer
Developer


Joined: 08 Aug 2002
Posts: 509

PostPosted: Mon Aug 30, 2004 8:26 pm    Post subject: Reply with quote

hehe, it seems to have a problem selecting the right file to diff from:

Code:
Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from
libxslt-1.1.8.tar.bz2
metalog-0.8_pre20031130.tar.gz
ncurses-5.4.tar.gz
net-snmp-5.1.1.tar.gz
openvpn-1.6.0.tar.gz
rc-scripts-1.5.2.tar.bz2
slocate-2.7.tar.gz
texinfo-4.7.tar.bz2
vim-6.3-lang.tar.gz

The best of all is ... vim-6.3-lang.tar.gz

Trying to download vim-6.3-lang.tar.gz-libxslt-1.1.9.tar.bz2.dtu

_________________
Queen Rocks.
Back to top
View user's profile Send private message
TPC
Tux's lil' helper
Tux's lil' helper


Joined: 16 Sep 2003
Posts: 135
Location: Sweden

PostPosted: Tue Aug 31, 2004 6:40 am    Post subject: Reply with quote

This needs to be fixed for slotted packages..
I had version 2.x of a slotted package, but then I tried to emerge a program that depended on 1.x, so it tried to get the 2.x to 1.x dtu (I aborted and downloaded manuall).

If it should do this or not, I'm not to decide (because I have no idea if this saves space or not), but it shouldn't delete the other version if you have set it to delete old files when new are created, for obvious reasons.
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Tue Aug 31, 2004 11:04 am    Post subject: Reply with quote

GurliGebis wrote:
hehe, it seems to have a problem selecting the right file to diff from:

Code:
Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from
libxslt-1.1.8.tar.bz2
metalog-0.8_pre20031130.tar.gz
ncurses-5.4.tar.gz
net-snmp-5.1.1.tar.gz
openvpn-1.6.0.tar.gz
rc-scripts-1.5.2.tar.bz2
slocate-2.7.tar.gz
texinfo-4.7.tar.bz2
vim-6.3-lang.tar.gz

The best of all is ... vim-6.3-lang.tar.gz

Trying to download vim-6.3-lang.tar.gz-libxslt-1.1.9.tar.bz2.dtu


What? 8O Which version of getdelta.sh are you using?
Is that patched? I know the alg. used to find the candidates sometimes fails but this example should never happen. What getdelta.sh does is:

1. remove any numbers and some special chars from the names
2. compare the results - and allow a maximum of 1 char differ
This point obviously failed in your example.
3. compare the candidates given by (2.) with exact filename (special-chars and numbers included) -- Again this failed in your example.

I've never seen that behaviour before (anyway the server would block a request for vim-6.3-lang.tar.gz-libxslt-1.1.9.tar.bz2.dtu as obviously INVALID.

I have seen false positives like trying to update libmcrypt from an existing libgcrypt, but *your* example does not match the rules above.

So you must have changed your getdeltas.sh manually - did you?

bp
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Tue Aug 31, 2004 11:14 am    Post subject: Reply with quote

pleusicles wrote:
First of all, thanks for this util, it's wonderful! :D

I just found a problem regarding the search for previously dl'ed distfiles (yes, I know it's tricky as stated in the script...):

I tried to upgrade mplayer, which includes downloading the following files:

font-arial-iso-8859-1.tar.bz2 and
font-arial-iso-8859-2.tar.bz2



Although this looks a bit funny - it's not wrong at all - these two files are similar enough to make a dtu - and actually you've saved 51.47% downloadsize.

And, NO, the MD5SUM is exactly that of the file the server got when created the patch. Deltup ensures this by including the md5sum of both - the source and the destination file in the dtu itself. And since you got an "OK" anything went right - according to deltup.

So why did you get an MD5SUM error then - probably the file has changed on mplayerhq.com - check bugs.gentoo.org if other users who do not use getdelta.sh have the same problem - I guess the MD5SUM stored in portage tree does not match the file from the mplayer's site anymore.

The only problem with the files above is:
If you would use getdelta.sh with REMOVE_OLD=true it would delete the "old" file -- and you would fail later since the file is missing. The only way to prevent this at the moment is to touch the files in your $DISTDIR. If they exist (even with 0 bytes) portage uses RESUMECOMMAND insteadt of FETCHCOMMAND - so it does not use getdelta.sh for the file and anything will be ok. (anyway, according to your output you did not use REMOVE_OLD, so this workaround is addressed to people who do)

bp
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Tue Aug 31, 2004 11:21 am    Post subject: Reply with quote

TPC wrote:
This needs to be fixed for slotted packages..
I had version 2.x of a slotted package, but then I tried to emerge a program that depended on 1.x, so it tried to get the 2.x to 1.x dtu (I aborted and downloaded manuall).


Downgrades are ok with deltup, too. Often enough it still saves downloadsize, as long as there are enough parts of the source-code that did not change.

However you do not want these files deleted, and the only way to prevent this at the moment is to touch the wanted filename in your distfiles directory. When portage finds the touched file it wont use getdelta.sh, but use your RESUMECOMMAND instead.

This is a dirty workaround, I know. I will include a feature with a user-defined exception-list where users can put in filenames they dont want to be deleted by the REMOVE_OLD feature. But this is a future-tune -- so use the the workaround until then.

Thanks.
bp
Back to top
View user's profile Send private message
MaxDamage
l33t
l33t


Joined: 03 Jan 2004
Posts: 650
Location: Oviedo, Spain

PostPosted: Tue Aug 31, 2004 12:47 pm    Post subject: Reply with quote

It's working very well here. Only tried three emerges, time will say if it works well 100%.
_________________
La PDA de tungsteno
Back to top
View user's profile Send private message
GurliGebis
Developer
Developer


Joined: 08 Aug 2002
Posts: 509

PostPosted: Tue Aug 31, 2004 2:55 pm    Post subject: Reply with quote

Haven't changed my getdelta.sh .
Used the quied step by step, so I have no clue why this happends, but I'll update getdelta.sh when the next version comes out.
_________________
Queen Rocks.
Back to top
View user's profile Send private message
pleusicles
n00b
n00b


Joined: 26 May 2004
Posts: 44

PostPosted: Tue Aug 31, 2004 8:03 pm    Post subject: Reply with quote

Quote:
Although this looks a bit funny - it's not wrong at all - these two files are similar enough to make a dtu - and actually you've saved 51.47% downloadsize.


Yes, but the script downloaded the dtu 5 times :wink:

blackpenguin wrote:

So why did you get an MD5SUM error then - probably the file has changed on mplayerhq.com - check bugs.gentoo.org if other users who do not use getdelta.sh have the same problem - I guess the MD5SUM stored in portage tree does not match the file from the mplayer's site anymore.


I checked it; md5sum seems to be correct in portage. BTW, the files are downloaded correctly and verified if getdelta.sh is disabled. However, the problem is reproducable when using getdelta.sh

It would be interesting to know if others who DO use getdelta.sh have the same problem; so, if anyone is interested, please
Code:
emerge -f =media-video/mplayer-1.0_pre4-r7


... with no font-arial-iso-8859-*.tar.bz2 in DISTFILES.

Thanks in advance,
Pleusicles
Back to top
View user's profile Send private message
tomaw
Guru
Guru


Joined: 26 Mar 2003
Posts: 429
Location: UK

PostPosted: Tue Aug 31, 2004 8:13 pm    Post subject: Reply with quote

Mine works fine here fetching mplayer, here's the output:
Code:

tom ~ # rm /usr/portage-distfiles/font-arial-iso-8859-*
tom ~ # vi /etc/make.conf
tom ~ # emerge -f =media-video/mplayer-1.0_pre4-r7 --oneshot
Calculating dependencies ...done!
>>> emerge (1 of 1) media-video/mplayer-1.0_pre4-r7 to /
>>> Previously fetched file: Blue-1.4.tar.bz2 MD5 ;-)
>>> Downloading ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/distfiles/font-arial-iso-8859-1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage-distfiles

No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/distfiles/font-arial-iso-8859-1.tar.bz2: Empty host.
>>> Downloading ftp://mirrors.sec.informatik.tu-darmstadt.de/gentoo/distfiles/font-arial-iso-8859-1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage-distfiles

No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

--21:10:37--  ftp://mirrors.sec.informatik.tu-darmstadt.de/gentoo/distfiles/font-arial-iso-8859-1.tar.bz2
           => `font-arial-iso-8859-1.tar.bz2'
Resolving mirrors.sec.informatik.tu-darmstadt.de... 130.83.164.194
Connecting to mirrors.sec.informatik.tu-darmstadt.de[130.83.164.194]:21... failed: Connection refused.
>>> Downloading http://ftp.snt.utwente.nl/pub/os/linux/gentoo/distfiles/font-arial-iso-8859-1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

Chosen Mirror is gentoo

Searching for a previosly downloaded file in /usr/portage-distfiles

No old version of the requested file found.

The dtu could not be fetched, downloading full file from original URL

--21:10:39--  http://ftp.snt.utwente.nl/pub/os/linux/gentoo/distfiles/font-arial-iso-8859-1.tar.bz2
           => `font-arial-iso-8859-1.tar.bz2'
Resolving ftp.snt.utwente.nl... 130.89.175.1, 2001:610:1908:8000::175:1
Connecting to ftp.snt.utwente.nl[130.89.175.1]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 234,242 [application/x-tar]

100%[==========================================================================================================================>] 234,242      171.74K/s

21:10:42 (171.18 KB/s) - `font-arial-iso-8859-1.tar.bz2' saved [234242/234242]

>>> font-arial-iso-8859-1.tar.bz2 MD5 ;-)
>>> Previously fetched file: svgalib_helper-1.9.17-mplayer.tar.bz2 MD5 ;-)
>>> Downloading ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/distfiles/MPlayer-1.0pre4.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage-distfiles

We have following candidates to choose from
MPlayer-1.0pre5.tar.bz2

The best of all is ... MPlayer-1.0pre5.tar.bz2

Trying to download MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.dtu

--21:10:42--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=MPlayer-1.0pre5.tar.bz2&want=MPlayer-1.0pre4.tar.bz2&version=0.5.2
           => `deltup.php?have=MPlayer-1.0pre5.tar.bz2&want=MPlayer-1.0pre4.tar.bz2&version=0.5.2'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://linux01.gwdg.de/~nlissne/deltas/MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed [following]
--21:10:42--  http://linux01.gwdg.de/%7Enlissne/deltas/MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed
           => `MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed'
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45 [application/x-bzip2]

100%[==========================================================================================================================>] 45            --.--K/s

21:10:42 (439.45 KB/s) - `MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed' saved [45/45]

GOT MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed

$
{RED}The server could not build the dtu-file for MPlayer-1.0pre4.tar.bz2

reason:

sorry, could not get MPlayer-1.0pre5.tar.bz2


The dtu could not be fetched, downloading full file from original URL

ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/distfiles/MPlayer-1.0pre4.tar.bz2: Empty host.
>>> Downloading ftp://mirrors.sec.informatik.tu-darmstadt.de/gentoo/distfiles/MPlayer-1.0pre4.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage-distfiles

We have following candidates to choose from
MPlayer-1.0pre5.tar.bz2

The best of all is ... MPlayer-1.0pre5.tar.bz2

Trying to download MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.dtu

--21:10:46--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=MPlayer-1.0pre5.tar.bz2&want=MPlayer-1.0pre4.tar.bz2&version=0.5.2
           => `deltup.php?have=MPlayer-1.0pre5.tar.bz2&want=MPlayer-1.0pre4.tar.bz2&version=0.5.2'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://linux01.gwdg.de/~nlissne/deltas/MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed [following]
--21:10:46--  http://linux01.gwdg.de/%7Enlissne/deltas/MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed
           => `MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed'
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45 [application/x-bzip2]

100%[==========================================================================================================================>] 45            --.--K/s

21:10:46 (439.45 KB/s) - `MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed' saved [45/45]

GOT MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed

$
{RED}The server could not build the dtu-file for MPlayer-1.0pre4.tar.bz2

reason:

sorry, could not get MPlayer-1.0pre5.tar.bz2


The dtu could not be fetched, downloading full file from original URL

--21:10:46--  ftp://mirrors.sec.informatik.tu-darmstadt.de/gentoo/distfiles/MPlayer-1.0pre4.tar.bz2
           => `MPlayer-1.0pre4.tar.bz2'
Resolving mirrors.sec.informatik.tu-darmstadt.de... 130.83.164.194
Connecting to mirrors.sec.informatik.tu-darmstadt.de[130.83.164.194]:21... failed: Connection refused.
>>> Downloading http://ftp.snt.utwente.nl/pub/os/linux/gentoo/distfiles/MPlayer-1.0pre4.tar.bz2
Checking if a Thirdparty-Mirror is used ...

Chosen Mirror is gentoo

Searching for a previosly downloaded file in /usr/portage-distfiles

We have following candidates to choose from
MPlayer-1.0pre5.tar.bz2

The best of all is ... MPlayer-1.0pre5.tar.bz2

Trying to download MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.dtu

--21:10:47--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=MPlayer-1.0pre5.tar.bz2&want=MPlayer-1.0pre4.tar.bz2&version=0.5.2&mirror=gentoo&path=/
           => `deltup.php?have=MPlayer-1.0pre5.tar.bz2&want=MPlayer-1.0pre4.tar.bz2&version=0.5.2&mirror=gentoo&path=%2F'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://linux01.gwdg.de/~nlissne/deltas/MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed [following]
--21:10:47--  http://linux01.gwdg.de/%7Enlissne/deltas/MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed
           => `MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed'
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 45 [application/x-bzip2]

100%[==========================================================================================================================>] 45            --.--K/s

21:10:47 (439.45 KB/s) - `MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed' saved [45/45]

GOT MPlayer-1.0pre5.tar.bz2-MPlayer-1.0pre4.tar.bz2.failed

$
{RED}The server could not build the dtu-file for MPlayer-1.0pre4.tar.bz2

reason:

sorry, could not get MPlayer-1.0pre5.tar.bz2


The dtu could not be fetched, downloading full file from original URL

--21:10:47--  http://ftp.snt.utwente.nl/pub/os/linux/gentoo/distfiles/MPlayer-1.0pre4.tar.bz2
           => `MPlayer-1.0pre4.tar.bz2'
Resolving ftp.snt.utwente.nl... 130.89.175.1, 2001:610:1908:8000::175:1
Connecting to ftp.snt.utwente.nl[130.89.175.1]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4,913,390 [application/x-tar]

100%[==========================================================================================================================>] 4,913,390    123.39K/s    ETA 00:00

21:11:17 (162.69 KB/s) - `MPlayer-1.0pre4.tar.bz2' saved [4913390/4913390]

>>> MPlayer-1.0pre4.tar.bz2 MD5 ;-)
>>> Downloading ftp:///ftp-stud.fht-esslingen.de/pub/Mirrors/gentoo/distfiles/font-arial-iso-8859-2.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage-distfiles

We have following candidates to choose from
font-arial-iso-8859-1.tar.bz2

The best of all is ... font-arial-iso-8859-1.tar.bz2

Trying to download font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu

_________________
Tom Wesley
Back to top
View user's profile Send private message
pleusicles
n00b
n00b


Joined: 26 May 2004
Posts: 44

PostPosted: Tue Aug 31, 2004 9:52 pm    Post subject: Reply with quote

tomaw: thanks for trying. The only difference I see is that - for some reason unknown to me - in your case font-arial-iso-8859-1.tar.bz2 is downloaded prior to -2.tar.bz2, while on my machine the reverse is true.

I tried replicate your case by
1. fetching mplayer with getdelta.sh disabled; both font files were downloaded successfully.
2. Then I deleted -2.tar.bz2 only,
3. then enabled getdelta.sh and did an emerge -f again:

Code:

>>> emerge (1 of 1) media-video/mplayer-1.0_pre4-r7 to /
>>> Previously fetched file: svgalib_helper-1.9.17-mplayer.tar.bz2 MD5 ;-)
>>> Downloading http://gentoo.osuosl.org/distfiles/font-arial-iso-8859-2.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /store/distfiles

We have following candidates to choose from
font-arial-iso-8859-1.tar.bz2

The best of all is ... font-arial-iso-8859-1.tar.bz2

Trying to download font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu

--23:22:20--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=font-arial-iso-8859-1.tar.bz2&want=font-arial-iso-8859-2.tar.bz2&version=0.5.2
           => `deltup.php?have=font-arial-iso-8859-1.tar.bz2&want=font-arial-iso-8859-2.tar.bz2&version=0.5.2'
Resolving linux01.gwdg.de... 134.76.13.21
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://linux01.gwdg.de/~nlissne/deltas/font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu [following]
--23:22:20--  http://linux01.gwdg.de/%7Enlissne/deltas/font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu
           => `font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu'
Connecting to linux01.gwdg.de[134.76.13.21]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104,815 [application/x-bzip2]

100%[=============================================================>] 104,815       89.16K/s

23:22:21 (88.90 KB/s) - `font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu' saved [104815/104815]

GOT font-arial-iso-8859-1.tar.bz2-font-arial-iso-8859-2.tar.bz2.dtu

Yes, we got the dtu-file now let's build font-arial-iso-8859-2.tar.bz2...

font-arial-iso-8859-1.tar.bz2 -> font-arial-iso-8859-2.tar.bz2: previous package is corrupt
cleaning up
The dtu could not be fetched, downloading full file from original URL

--23:22:22--  http://gentoo.osuosl.org/distfiles/font-arial-iso-8859-2.tar.bz2
           => `font-arial-iso-8859-2.tar.bz2'
Resolving gentoo.osuosl.org... 140.211.166.134
Connecting to gentoo.osuosl.org[140.211.166.134]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 222,208 [application/x-tar]

100%[=============================================================>] 222,208       64.52K/s    ETA 00:00

23:22:26 (64.37 KB/s) - `font-arial-iso-8859-2.tar.bz2' saved [222208/222208]

>>> font-arial-iso-8859-2.tar.bz2 MD5 ;-)
>>> Previously fetched file: font-arial-iso-8859-1.tar.bz2 MD5 ;-)
>>> Previously fetched file: mplayer-1.0_pre4-mga-kernel2.6.patch.tar.bz2 MD5 ;-)
>>> Previously fetched file: MPlayer-1.0pre4.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) MPlayer-1.0pre4.tar.bz2
>>> md5 src_uri ;-) font-arial-iso-8859-1.tar.bz2
>>> md5 src_uri ;-) font-arial-iso-8859-2.tar.bz2
>>> md5 src_uri ;-) mplayer-1.0_pre4-mga-kernel2.6.patch.tar.bz2
>>> md5 src_uri ;-) svgalib_helper-1.9.17-mplayer.tar.bz2


It seems that portage accepted the md5sum of -1.tar.bz2, but deltup found the same file to be corrupt
("previous package is corrupt"). Interesting...
Back to top
View user's profile Send private message
mikenerone
n00b
n00b


Joined: 11 Feb 2004
Posts: 21
Location: San Antonio, TX

PostPosted: Wed Sep 01, 2004 6:43 am    Post subject: Reply with quote

blackpenguin wrote:
riksta wrote:
forgive me if i am wrong, but why not just use RSYNC for this?

Uhm.... what? I am not sure if I catch your idea. - If you mean to unpack the local old version to somewhere and then rsync with the wanted version - well - that cannot be done with a downloadwrapper since you will fail to get the same MD5SUM again when compressing back to an archive again.

I think riksta may have meant the same thing that occurred to me (but I, too, am not sure this would work the way I think): After finding a best candidate for a binary diff, couldn't portage (or a FETCHCOMMAND) simply copy that candidate to the new filename in distfiles, then "update" that file via rsync from a distfile mirror?
Back to top
View user's profile Send private message
PrakashP
Veteran
Veteran


Joined: 27 Oct 2003
Posts: 1246
Location: C.C.A.A., Germania

PostPosted: Wed Sep 01, 2004 7:13 am    Post subject: Reply with quote

Grrr, I still have the problem that the server doesn't find files to generate the dtu.

In my case it was with imlib2-1.1.1 to imlib2-1.1.2 and with ut2004.

Code:

Calculating dependencies ...done!
>>> emerge (1 of 1) games-fps/ut2004-3270 to /
>>> Downloading http://mirror1.icculus.org/ut2004/ut2004-lnxpatch3270-1.tar.bz2
Checking if a Thirdparty-Mirror is used ...

No third-party mirror.

Searching for a previosly downloaded file in /usr/portage/distfiles

We have following candidates to choose from
ut2004-lnxpatch3236-1.tar.bz2

The best of all is ... ut2004-lnxpatch3236-1.tar.bz2

Trying to download ut2004-lnxpatch3236-1.tar.bz2-ut2004-lnxpatch3270-1.tar.bz2.dtu

--09:09:51--  http://linux01.gwdg.de/%7Enlissne/deltup.php?have=ut2004-lnxpatch3236-1.tar.bz2&want=ut2004-lnxpatch3270-1.tar.bz2&version=0.5.2
           => `deltup.php?have=ut2004-lnxpatch3236-1.tar.bz2&want=ut2004-lnxpatch3270-1.tar.bz2&version=0.5.2'
Auflösen des Hostnamen »linux01.gwdg.de«.... 134.76.13.21
Verbindungsaufbau zu linux01.gwdg.de[134.76.13.21]:80... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 302 Found
Platz: http://linux01.gwdg.de/~nlissne/deltas/ut2004-lnxpatch3236-1.tar.bz2-ut2004-lnxpatch3270-1.tar.bz2.failed[folge]
--09:09:51--  http://linux01.gwdg.de/%7Enlissne/deltas/ut2004-lnxpatch3236-1.tar.bz2-ut2004-lnxpatch3270-1.tar.bz2.failed
           => `ut2004-lnxpatch3236-1.tar.bz2-ut2004-lnxpatch3270-1.tar.bz2.failed'
Verbindungsaufbau zu linux01.gwdg.de[134.76.13.21]:80... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 51 [application/x-bzip2]

100%[==================================================================>] 51            --.--K/s

09:09:51 (498.05 KB/s) - »ut2004-lnxpatch3236-1.tar.bz2-ut2004-lnxpatch3270-1.tar.bz2.failed« gespeichert [51/51]

GOT ut2004-lnxpatch3236-1.tar.bz2-ut2004-lnxpatch3270-1.tar.bz2.failed


The server could not build the dtu-file for ut2004-lnxpatch3270-1.tar.bz2

reason:

sorry, could not get ut2004-lnxpatch3236-1.tar.bz2


The dtu could not be fetched, downloading full file from original URL

--09:09:51--  http://mirror1.icculus.org/ut2004/ut2004-lnxpatch3270-1.tar.bz2
           => `ut2004-lnxpatch3270-1.tar.bz2'
Auflösen des Hostnamen »mirror1.icculus.org«.... 216.239.132.108
Verbindungsaufbau zu mirror1.icculus.org[216.239.132.108]:80... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 19,664,589 [application/octet-stream]

 0% [                                                                   ] 20,112        26.36K/s             


I checked it and the older version of ut2004 is downladable at that location, so I wonder why the server isn't able to do so.

BTW, I set GENTOO_MIRRORS="".
Back to top
View user's profile Send private message
Nominus
n00b
n00b


Joined: 15 Feb 2004
Posts: 17

PostPosted: Wed Sep 01, 2004 8:49 pm    Post subject: Reply with quote

I have noticed that detlup producing algoritm isn't enouth efficient as it could be, for example today the server failed to produce size-efficient delta for 5 files, and i had to download xdeltas from http://gd.tuwien.ac.at/opsys/linux/sorcerer/xdeltas/ which are still size-efficient with same files.
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Wed Sep 01, 2004 11:12 pm    Post subject: Reply with quote

Nominus wrote:
I have noticed that detlup producing algoritm isn't enouth efficient as it could be, for example today the server failed to produce size-efficient delta for 5 files, and i had to download xdeltas from http://gd.tuwien.ac.at/opsys/linux/sorcerer/xdeltas/ which are still size-efficient with same files.


You should know that deltup is just a wrapper for xdelta-files - actually the difference is done by xdelta. Therefore it's hard to believe what you say. (but it's ok, if you just wanted to give another source of delta-files - we never can have enough of good links)

You should also see, that deltup is md5-aware - that means you do not only get a file with the same archive-content - it's bit-identical - which it is not with xdelta only - at least not when used with *.gz files. But we need a bit-identical file to pass the MD5 check of portage (and actually to be sure the file haven't changed on the way)

However, feel *free* to use another source of deltas - just do what you feel what you need to do.

bp
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Thu Sep 02, 2004 12:12 am    Post subject: Reply with quote

PrakashKC wrote:
Grrr, I still have the problem that the server doesn't find files to generate the dtu.

In my case it was with imlib2-1.1.1 to imlib2-1.1.2 and with ut2004.

...

I checked it and the older version of ut2004 is downladable at that location, so I wonder why the server isn't able to do so.

BTW, I set GENTOO_MIRRORS="".


about imlib2-1.1.1 to imlib2-1.1.2 - I wonder what you mean, since there is a dtu for that on the server,

about ut2004 - as you might have seen in your output, it says "No third party mirror" - what that means is it didn't find any mirror-group for your URL - and the server tries to download it from a list of known gentoo-mirrors then. But they do not mirror ut2004 -- you cannot download arbitrary files via the deltup-server.

getdelta.sh tries to find the correct mirror-group (therefore it makes sense to set GENTOO_MIRRORS="") and gives the server a clue where to find the wanted file with that. ut2004 is not available on any of the known mirrors - neither the gentoo mirrors, nor any of the third-party-mirrors.
So that's the reason why the server could not find the file (the original URL is not passed to the server - and it would not make sense to do so)

bp
Back to top
View user's profile Send private message
blackpenguin
n00b
n00b


Joined: 09 Mar 2004
Posts: 43
Location: Germany

PostPosted: Thu Sep 02, 2004 12:38 am    Post subject: Reply with quote

pleusicles wrote:

It seems that portage accepted the md5sum of -1.tar.bz2, but deltup found the same file to be corrupt
("previous package is corrupt"). Interesting...


Neither portage nor deltup is wrong in that case. As you can check on bugs.gentoo.org - the font-files on mplayerhq change every few months without any obvious reason - therefore the md5sum often changes.
"previous package is corrupt" means, the file the server got from mplayerhq is not the one you had in your distfiles already, but you have the one, that the package-maintainer has, too.

I dont know why the mplayer-team change these files from time to time (this might be just an untar and tar again) - but they do - it happens again and again that the portage-md5sum does not match especially for these 2 files. Not much we can do about this.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3, ... 9, 10, 11  Next
Page 2 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum