Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO - Parallel Portage: Speed up the long emerges
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Thu Sep 07, 2006 4:18 pm    Post subject: HOWTO - Parallel Portage: Speed up the long emerges Reply with quote

Please follow this link to get the download and instructions. I originally posted there to start the discussions about an idea, which materialized as a patch.

http://forums.gentoo.org/viewtopic-t-484842-start-0.html

The changes that you will see and hopefully like are:

1. Parallel install of certain packages keeping your cpu pegged. note that it may not always be possible e.g. you want to emerge -O kdelibs kdebase...:)

2. The output is very concise if PORT_LOGDIR is set. Only important messages (like setup and post install messages which are currently lost in the garbage that flies by) are shown in the terminal, rest go to log files in /var/log/portage. I highly recommend setting it. If you don't, your configure/compile/install garbage will be thrown on the terminal as before. Less output on the terminal speeds up emerges a lot.

3. If a package fails, emerge doesn't stop immdiately. It will keep emerging as long as there are packages in the current slot to merge. It will stop after the current slot. A slot is like a level in the depgraph. All leaf nodes (packages which don't depend on anything) are at slot 0 (or 1). Next level packages are at slot 1, and so on. All packages at the same slot are not related to each other and can be emerged at the same time. In nutshell, portage will stop only when its absolutely impossible to proceed with any emerge.

Speed gains of more than 25% have been observed. YMMV depending upon what you emerge.

Now you actually can see the whole output of 'emerge -e system' in your terminal, and note down what all stuff you need to do manually.


Last edited by devsk on Fri Sep 08, 2006 6:13 pm; edited 1 time in total
Back to top
View user's profile Send private message
irwinr
Apprentice
Apprentice


Joined: 11 May 2004
Posts: 152
Location: Texas

PostPosted: Fri Sep 08, 2006 4:51 am    Post subject: Reply with quote

When can we expect to see this merged into portage?

-Jeremy
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Fri Sep 08, 2006 5:22 am    Post subject: Reply with quote

when we have cleaned the code and fixed whatever issues we find...;-)
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Fri Sep 08, 2006 7:41 am    Post subject: Reply with quote

Uploaded new tar and a new patch file.

Code:
Brief Changelog:
          handle pkg failure and exit better.
          clean up unmerge.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Fri Sep 08, 2006 6:10 pm    Post subject: Reply with quote

Fixed another bug, wherein even 'nomerge' packages were getting slotted, in latest version.
Back to top
View user's profile Send private message
Zentoo
Apprentice
Apprentice


Joined: 18 Nov 2002
Posts: 171
Location: /dev/console

PostPosted: Sun Sep 10, 2006 7:02 pm    Post subject: First tests with parallel patch applied ... Reply with quote

I apply the patch since it's for a amd64 box. I use portage-2.1.1 (where is the -rc4 ?) on a ~amd64 profile and two patch's chunks failed so i open my editor and finish to patch manually. Obviously i quickpkg'ed portage before.

My first test:

Code:
  ###################################
 #### Without Parallel Feature #####
###################################

time nice -n -20 MAKEOPTS="-j3" emerge  gftp gcalctool gedit gnome-system-monitor gnome-alsamixer grip
real    3m14.987s
user    2m23.299s
sys     1m34.905s

  ################################
 #### With Parallel Feature #####
################################

time FEATURES="parallel" MAKEOPTS="-j3" nice -n -20 emerge  gftp gcalctool gedit gnome-system-monitor gnome-alsamixer grip
real    2m13.374s
user    2m29.092s
sys     1m42.797s

so that's a real big improvement.
time-real-no-parallel / time-real-parallel = 0.684 so 31.6% time less :!:

After that i wonder what was the real time gain due just to output redirection so i decided to backup my portage using quickpkg and cp/mv in packages directory to be sure i could emerge the parallel and the no-parallel portage easily back.

So i start to put my old portage back with an "emerge -avk portage" and HO HO surprise :!:

Code:
# emerge -avk portage

These are the packages that would be merged, in order:

Calculating dependencies... done!
Package list for slot = 0
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']
[binary   R   ] sys-apps/portage-2.1.1  USE="-build -doc (-selinux)" LINGUAS="-pl"

Total size of downloads: 0 kB
Package list for slot = 0
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']

Would you like to merge these packages? [Yes/No]
Package list for slot = 0
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']

>>> Emerging (1 of 1) sys-apps/portage-2.1.1 to /

>>> Emerging (2 of 1) sys-apps/portage-2.1.1 to /

>>> Emerging (3 of 1) sys-apps/portage-2.1.1 to /
>>> Extracting info
Traceback (most recent call last):
  File "/usr/bin/emerge", line 4224, in ?
    emerge_main()
  File "/usr/bin/emerge", line 4219, in emerge_main
    myopts, myaction, myfiles, spinner)
  File "/usr/bin/emerge", line 3759, in action_build
    mergetask.merge(pkglist, favorites, mtimedb)
  File "/usr/bin/emerge", line 2289, in merge
    onepid = self.fork_one_emerge(y,mergecount,len(mymergelist),mtimedb,favorites,mysysdict)
  File "/usr/bin/emerge", line 2032, in fork_one_emerge
    prev_mtimes=ldpath_mtimes)
  File "/usr/lib/portage/pym/portage.py", line 6538, in pkgmerge
    os.chdir(pkgloc)
OSError: [Errno 2] No such file or directory: '/gentoo/tmp/binpkgs//portage-2.1.1/bin/'
Traceback (most recent call last):
  File "/usr/bin/emerge", line 4224, in ?
    emerge_main()
  File "/usr/bin/emerge", line 4219, in emerge_main
    myopts, myaction, myfiles, spinner)
  File "/usr/bin/emerge", line 3759, in action_build
    mergetask.merge(pkglist, favorites, mtimedb)
  File "/usr/bin/emerge", line 2289, in merge
    onepid = self.fork_one_emerge(y,mergecount,len(mymergelist),mtimedb,favorites,mysysdict)
  File "/usr/bin/emerge", line 2032, in fork_one_emerge
    prev_mtimes=ldpath_mtimes)
  File "/usr/lib/portage/pym/portage.py", line 6532, in pkgmerge
    os.makedirs(infloc)
  File "/usr/lib64/python2.4/os.py", line 159, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/gentoo/tmp/binpkgs//portage-2.1.1/inf/'
>>> Extracting info
>>> Extracting portage-2.1.1
tar: ./etc/portage/.keep_sys-apps_portage-0: Cannot open: Not a directory
tar: ./usr/lib/portage: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin/archive-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/chkcontents: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/clean_locks: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dispatch-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dobin: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doconfd: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dodir: Cannot open: No such file or directory
tar: Skipping to next header

  [i]---  few hundred of lines have been cut here since that's similar lines ---[/i]

tar: ./usr/lib/portage/pym/portage_exec.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_gpg.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_localization.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_locks.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_mail.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_manifest.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_selinux.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_update.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_util.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_versions.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/xpak.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: Error exit delayed from previous errors
!!! Error Extracting /gentoo/packages/All/portage-2.1.1.tbz2
/usr/lib/portage/bin/ebuild.sh: line 1454: /gentoo/tmp/binpkgs/portage-2.1.1/inf/portage-2.1.1.ebuild: No such file or directory

!!! ERROR: sys-apps/portage-2.1.1 failed.
Call stack:
  ebuild.sh, line 1454:   Called die

!!! error sourcing ebuild
!!! If you need support, post the topmost build error, and the call stack if relevant.

/usr/lib/portage/bin/ebuild.sh: line 1454: /gentoo/tmp/binpkgs/portage-2.1.1/inf/portage-2.1.1.ebuild: No such file or directory

!!! ERROR: sys-apps/portage-2.1.1 failed.
Call stack:
  ebuild.sh, line 1454:   Called die

!!! error sourcing ebuild
!!! If you need support, post the topmost build error, and the call stack if relevant.

Package sys-apps/portage-2.1.1 failed to emerge
Please take a look at the file /gentoo/log/sys-apps:portage-2.1.1:20060910-183829.log


So I don't know how continue since the only thing i could do seems to emerge portage from source back losting the patched version on the way. And i don't want to finish patch by hand each time.

1) How could i do to make another diff to have a clean patch to aply on my portage ? :roll:

2) Why the binary emerge have failed ? I try to use quipkg/emerge -k on another pkg and that was ok so i don't understand really . 8O



I have bench the last nigh an emerge -e system and should try with the parallel feature this night to write down here time comparaison.
_________________
Kernel 3.14.6| Gcc 4.8.3 | Glibc 2.19
Core i7 990x @ 4.0GHz | X58 | 12Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Sun Sep 10, 2006 7:44 pm    Post subject: Reply with quote

something really bizzare happened here. I am still trying to analyse it. I will let you know soon.

Another thing is that before emerge -e system, you should make sure that it won't emerge portage itself because in that case, it will overwrite portage midway of emerge -e system and screw ups will happen. easiest way is to put the current tree version into /etc/portage/profile/package.provided (which unfortunately will need to be modified if you did --sync again and a newer portage version is available).

PS: There is a problem with the --ask codepath. Don't use --ask for now. Use --pretend and then emerge. I will have the fix ready soon.

PPS:The issue with --ask, wherein it tries three emerges for the same package, is fixed in latest tar.


Last edited by devsk on Mon Sep 11, 2006 8:43 pm; edited 1 time in total
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Sun Sep 10, 2006 7:50 pm    Post subject: Reply with quote

@Zentoo, please do the same emerge without --ask and let me know if it fails. You should be able to toggle between parallel portage and normal portage with this approach.
Back to top
View user's profile Send private message
Zentoo
Apprentice
Apprentice


Joined: 18 Nov 2002
Posts: 171
Location: /dev/console

PostPosted: Sun Sep 10, 2006 8:18 pm    Post subject: Check for ask usage on emerge with parallel patch applied Reply with quote

So i test it and that make almost the same thing...

Code:
#emerge -pvk portage

These are the packages that would be merged, in order:

Calculating dependencies... done!
Package list for slot = 0
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']
[binary   R   ] sys-apps/portage-2.1.1  USE="-build -doc (-selinux)" LINGUAS="-pl"

Total size of downloads: 0 kB


Code:
# emerge -k portage
Calculating dependencies... done!
Package list for slot = 0
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']

>>> Emerging (1 of 1) sys-apps/portage-2.1.1 to /
>>> Extracting info
>>> Extracting portage-2.1.1
tar: ./etc/portage/.keep_sys-apps_portage-0: Cannot open: Not a directory
tar: ./usr/lib/portage: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin/archive-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/chkcontents: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/clean_locks: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dispatch-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dobin: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doconfd: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dodir: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dodoc: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doenvd: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doexe: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dohard: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dohtml: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doinfo: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doinitd: Cannot open: No such file or directory

[i]   ----  hundred of lines similar ---------------[/i]

tar: ./usr/lib/portage/pym/portage_locks.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_mail.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_manifest.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_selinux.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_update.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_util.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_versions.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/xpak.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: Error exit delayed from previous errors
!!! Error Extracting /gentoo/packages/All/portage-2.1.1.tbz2
/usr/lib/portage/bin/ebuild.sh: line 1454: /gentoo/tmp/binpkgs/portage-2.1.1/inf/portage-2.1.1.ebuild: No such file or directory

!!! ERROR: sys-apps/portage-2.1.1 failed.
Call stack:
  ebuild.sh, line 1454:   Called die

!!! error sourcing ebuild
!!! If you need support, post the topmost build error, and the call stack if relevant.

/usr/lib/portage/bin/ebuild.sh: line 1454: /gentoo/tmp/binpkgs/portage-2.1.1/inf/portage-2.1.1.ebuild: No such file or directory

!!! ERROR: sys-apps/portage-2.1.1 failed.
Call stack:
  ebuild.sh, line 1454:   Called die

!!! error sourcing ebuild
!!! If you need support, post the topmost build error, and the call stack if relevant.

Package sys-apps/portage-2.1.1 failed to emerge
Please take a look at the file /gentoo/log/sys-apps:portage-2.1.1:20060910-183829.log


Something interesting may be that my portage log directory have strange new entry that don't start with a number anymore with this 2.1.1 version here (i noticed it before use the parallel patch).

En extract from "ls -altr"
Code:
-rw-r--r--  1 root    portage   129399 Sep 10 18:47 gnome-extra:gcalctool-5.8.20:20060910-164703.log
-rw-r--r--  1 root    portage    56894 Sep 10 18:48 net-ftp:gftp-2.0.18-r4:20060910-164703.log
-rw-r--r--  1 root    portage    19377 Sep 10 18:48 media-sound:gnome-alsamixer-0.9.6:20060910-164810.log
-rw-r--r--  1 root    portage      549 Sep 10 18:48 gnome-extra:gnome-system-monitor-2.14.4:20060910-164841.log
-rw-r--r--  1 root    portage    94754 Sep 10 18:48 gnome-extra:gnome-system-monitor-2.14.4:20060910-164756.log
-rw-r--r--  1 root    portage    57735 Sep 10 18:49 media-sound:grip-3.3.1:20060910-164828.log
-rw-r--r--  1 root    portage      549 Sep 10 18:49 app-editors:gedit-2.14.4:20060910-164909.log
-rw-r--r--  1 root    portage   293312 Sep 10 18:49 app-editors:gedit-2.14.4:20060910-164703.log
-rw-r--r--  1 root    portage      549 Sep 10 18:50 gnome-extra:gcalctool-5.8.20:20060910-165029.log
-rw-r--r--  1 root    portage    56894 Sep 10 18:50 net-ftp:gftp-2.0.18-r4:20060910-164939.log
-rw-r--r--  1 root    portage   129399 Sep 10 18:51 gnome-extra:gcalctool-5.8.20:20060910-164939.log
-rw-r--r--  1 root    portage    19377 Sep 10 18:51 media-sound:gnome-alsamixer-0.9.6:20060910-165102.log
-rw-r--r--  1 root    portage      549 Sep 10 18:51 app-editors:gedit-2.14.4:20060910-165133.log
-rw-r--r--  1 root    portage   293326 Sep 10 18:51 app-editors:gedit-2.14.4:20060910-164939.log
-rw-r--r--  1 root    portage      549 Sep 10 18:51 gnome-extra:gnome-system-monitor-2.14.4:20060910-165143.log
-rw-r--r--  1 root    portage    94754 Sep 10 18:51 gnome-extra:gnome-system-monitor-2.14.4:20060910-165101.log
-rw-r--r--  1 root    portage    57735 Sep 10 18:51 media-sound:grip-3.3.1:20060910-165117.log
-rw-r--r--  1 root    portage      549 Sep 10 18:53 gnome-extra:gcalctool-5.8.20:20060910-165313.log
-rw-r--r--  1 root    portage   129399 Sep 10 18:53 gnome-extra:gcalctool-5.8.20:20060910-165232.log
-rw-r--r--  1 root    portage    56894 Sep 10 18:53 net-ftp:gftp-2.0.18-r4:20060910-165232.log
-rw-r--r--  1 root    portage      549 Sep 10 18:54 gnome-extra:gnome-system-monitor-2.14.4:20060910-165405.log
-rw-r--r--  1 root    portage    94754 Sep 10 18:54 gnome-extra:gnome-system-monitor-2.14.4:20060910-165320.log
-rw-r--r--  1 root    portage    19377 Sep 10 18:54 media-sound:gnome-alsamixer-0.9.6:20060910-165348.log
-rw-r--r--  1 root    portage      549 Sep 10 18:54 app-editors:gedit-2.14.4:20060910-165436.log
-rw-r--r--  1 root    portage   293326 Sep 10 18:54 app-editors:gedit-2.14.4:20060910-165232.log
-rw-r--r--  1 root    portage    57735 Sep 10 18:54 media-sound:grip-3.3.1:20060910-165411.log
-rw-r--r--  1 root    portage      549 Sep 10 18:56 gnome-extra:gcalctool-5.8.20:20060910-165638.log
-rw-r--r--  1 root    portage   129399 Sep 10 18:56 gnome-extra:gcalctool-5.8.20:20060910-165543.log
-rw-r--r--  1 root    portage    56894 Sep 10 18:56 net-ftp:gftp-2.0.18-r4:20060910-165543.log
-rw-r--r--  1 root    portage    19377 Sep 10 18:57 media-sound:gnome-alsamixer-0.9.6:20060910-165656.log
-rw-r--r--  1 root    portage      549 Sep 10 18:57 gnome-extra:gnome-system-monitor-2.14.4:20060910-165724.log
-rw-r--r--  1 root    portage    94754 Sep 10 18:57 gnome-extra:gnome-system-monitor-2.14.4:20060910-165648.log
-rw-r--r--  1 root    portage      549 Sep 10 18:57 app-editors:gedit-2.14.4:20060910-165749.log
-rw-r--r--  1 root    portage    57735 Sep 10 18:57 media-sound:grip-3.3.1:20060910-165713.log
-rw-r--r--  1 root    portage   293326 Sep 10 18:57 app-editors:gedit-2.14.4:20060910-165543.log
-rw-r--r--  1 root    portage    56894 Sep 10 19:04 net-ftp:gftp-2.0.18-r4:20060910-170401.log
-rw-r--r--  1 root    portage      549 Sep 10 19:04 gnome-extra:gcalctool-5.8.20:20060910-170457.log
-rw-r--r--  1 root    portage   129399 Sep 10 19:05 gnome-extra:gcalctool-5.8.20:20060910-170430.log
-rw-r--r--  1 root    portage      549 Sep 10 19:06 app-editors:gedit-2.14.4:20060910-170605.log
-rw-r--r--  1 root    portage   293326 Sep 10 19:06 app-editors:gedit-2.14.4:20060910-170501.log
-rw-r--r--  1 root    portage      549 Sep 10 19:06 gnome-extra:gnome-system-monitor-2.14.4:20060910-170635.log
-rw-r--r--  1 root    portage    94754 Sep 10 19:06 gnome-extra:gnome-system-monitor-2.14.4:20060910-170610.log
-rw-r--r--  1 root    portage    19377 Sep 10 19:06 media-sound:gnome-alsamixer-0.9.6:20060910-170639.log
-rw-r--r--  1 root    portage    57735 Sep 10 19:07 media-sound:grip-3.3.1:20060910-170649.log
-rw-r--r--  1 root    portage    56894 Sep 10 19:08 net-ftp:gftp-2.0.18-r4:20060910-170747.log
-rw-r--r--  1 root    portage      549 Sep 10 19:08 gnome-extra:gcalctool-5.8.20:20060910-170844.log
-rw-r--r--  1 root    portage   129399 Sep 10 19:08 gnome-extra:gcalctool-5.8.20:20060910-170817.log
-rw-r--r--  1 root    portage      549 Sep 10 19:09 app-editors:gedit-2.14.4:20060910-170952.log
-rw-r--r--  1 root    portage   293326 Sep 10 19:09 app-editors:gedit-2.14.4:20060910-170847.log
-rw-r--r--  1 root    portage      549 Sep 10 19:10 gnome-extra:gnome-system-monitor-2.14.4:20060910-171021.log
-rw-r--r--  1 root    portage    94754 Sep 10 19:10 gnome-extra:gnome-system-monitor-2.14.4:20060910-170957.log
-rw-r--r--  1 root    portage    19377 Sep 10 19:10 media-sound:gnome-alsamixer-0.9.6:20060910-171025.log
-rw-r--r--  1 root    portage    57735 Sep 10 19:10 media-sound:grip-3.3.1:20060910-171035.log
-rw-r--r--  1 root    portage      549 Sep 10 20:01 gnome-extra:gcalctool-5.8.20:20060910-180108.log
-rw-r--r--  1 root    portage   129399 Sep 10 20:01 gnome-extra:gcalctool-5.8.20:20060910-180041.log
-rw-r--r--  1 root    portage     5029 Sep 10 20:11 net-ftp:gftp-2.0.18-r4:20060910-181137.log
-rw-r--r--  1 root    portage     7483 Sep 10 20:11 gnome-extra:gcalctool-5.8.20:20060910-181137.log
-rw-r--r--  1 root    portage     5385 Sep 10 20:11 app-editors:gedit-2.14.4:20060910-181137.log
-rw-r--r--  1 root    portage     4625 Sep 10 20:16 net-ftp:gftp-2.0.18-r4:20060910-181641.log
-rw-r--r--  1 root    portage     5985 Sep 10 20:16 app-editors:gedit-2.14.4:20060910-181641.log
-rw-r--r--  1 root    portage      223 Sep 10 20:17 net-ftp:gftp-2.0.18-r4:20060910-181703.log
-rw-r--r--  1 root    portage     1388 Sep 10 20:19 dev-lang:php-5.1.6-r4:20060910-181909.log
-rw-r--r--  1 root    portage     8479 Sep 10 20:21 sys-apps:ethtool-5:20060910-182139.log
-rw-r--r--  1 root    portage     7643 Sep 10 20:21 sys-apps:ethtool-5:20060910-182147.log
-rw-r--r--  1 root    portage      321 Sep 10 20:22 sys-apps:ethtool-4:20060910-182200.log
-rw-r--r--  1 root    portage      978 Sep 10 20:22 sys-apps:ethtool-4:20060910-150558.log
-rw-r--r--  1 root    portage     1277 Sep 10 20:24 dev-php4:pecl-sqlite-1.0.3:20060910-150558.log
-rw-r--r--  1 root    portage      549 Sep 10 20:36 gnome-extra:gcalctool-5.8.20:20060910-183604.log
-rw-r--r--  1 root    portage     5456 Sep 10 20:36 gnome-extra:gcalctool-5.8.20:20060910-181641.log
drwxrws---  2 portage portage     1386 Sep 10 20:47 .
-rw-r--r--  1 root    portage     2161 Sep 10 20:47 net-ftp:gftp-2.0.18-r4:20060910-184748.log
-rw-r--r--  1 root    portage     1283 Sep 10 20:47 gnome-extra:gcalctool-5.8.20:20060910-184748.log
-rw-r--r--  1 root    portage     1243 Sep 10 20:47 app-editors:gedit-2.14.4:20060910-184748.log
-rw-r--r--  1 root    portage      672 Sep 10 20:50 sys-apps:portage-2.1.1:20060910-183829.log

is it a new normal behaviour of portage ?



NB: i get use to always use stable portage and binutils on my unstable box for convenience about news bugs and I emerged the unstable portage here to apply the patch.

NB2: I have completely delocalised my portage stuff on /gentoo using make.conf variables and some links but that's rock stable since a lot like it. Even from my ~x86 box to ~amd64 one migration.
_________________
Kernel 3.14.6| Gcc 4.8.3 | Glibc 2.19
Core i7 990x @ 4.0GHz | X58 | 12Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Sun Sep 10, 2006 8:31 pm    Post subject: Reply with quote

log directory behaviour is normal. It is new in 2.1.1.

What is:

Code:

ls -al /usr/portage/packages/All
ls -al /var/tmp/portage

df -k /var/tmp/portage
df -k /
file /usr/bin/tbz2tool
file /usr/lib/portage/bin/tbz2tool


I find it interesting that it couldn't untar the tbz2. Are you sure that you have 64-bit tbz2tool for amd64 in /usr/bin?

I just tested your test case here in my install and without --ask, it emerges fine. With --ask, it reproduces your problem. I found the bug and I will put out a new patch/tbz2 soon.
Back to top
View user's profile Send private message
Zentoo
Apprentice
Apprentice


Joined: 18 Nov 2002
Posts: 171
Location: /dev/console

PostPosted: Sun Sep 10, 2006 8:42 pm    Post subject: Reply with quote

My bash answers are:

Code:
# ls -al /var/tmp/portage
total 2
drwxrwxr-x 4 portage portage 4 Sep 10 22:31 .
drwxrwxrwt 4 root    root    4 Sep 10 17:05 ..
drwxrwxr-x 4 root    portage 5 Sep 10 22:31 portage-2.1.1

#ls -al /gentoo/packages/All
total 19142
drwxr-xr-x 2 root root      12 Sep 10 20:35 .
drwxr-xr-x 8 root root       8 Sep 10 20:35 ..
-rw-r--r-- 1 root root  359411 Sep 10 07:23 portage-2.1-r2.tbz2
-rw-r--r-- 1 root root  373774 Sep 10 22:06 portage-2.1.1.tbz2
-rw-r--r-- 1 root root  373774 Sep 10 19:02 portage-2.1.1.tbz2.noparallel
-rw-r--r-- 1 root root  373938 Sep 10 19:03 portage-2.1.1.tbz2.parallel
-rw-r--r-- 1 root root 7061755 Sep 10 17:00 python-2.4.3-r3.tbz2


Code:

# df -k /var/tmp/portage
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md/1             18561924  12787552   5774372  69% /

# df -k /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md/1             18561924  12787552   5774372  69% /

Code:

# file /usr/bin/tbz2tool
/usr/bin/tbz2tool: symbolic link to `../lib/portage/bin/tbz2tool'

# file /usr/lib/portage/bin/tbz2tool
/usr/lib/portage/bin/tbz2tool: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped



Hoo ! you edit your last message while i type this one ! lol ... ok so i stay in standbye for your patch ...
_________________
Kernel 3.14.6| Gcc 4.8.3 | Glibc 2.19
Core i7 990x @ 4.0GHz | X58 | 12Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Sun Sep 10, 2006 9:04 pm    Post subject: Reply with quote

can you please test on a different package e.g.
Code:
quickpkg less
cp /usr/bin/less /usr/bin/less.I-dont-want-to-be-left-without-less
FEATURES=parallel emerge -k less


I still can't figure out why it refuses to untar. Does
Code:
\rm -rf /var/tmp/portage/*
\rm -rf /var/tmp/binpkgs/*
followed by
Code:
emerge -k portage
help?
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Sun Sep 10, 2006 9:43 pm    Post subject: Reply with quote

Can you please try the latest ebuild tar that I put on the link? Please untar it into /usr/local/portage with:
Code:
cd /usr/local/portage
tar xjpf /tmp/portage-parallel-2.1.1-ebuild.tar.bz2

make sure overlay is set to /usr/local/portage. emerge portage. This will give you a clean portage 2.1.1 with parallel patch in. Then, try to emerge the original no-parallel tbz2.
Back to top
View user's profile Send private message
Zentoo
Apprentice
Apprentice


Joined: 18 Nov 2002
Posts: 171
Location: /dev/console

PostPosted: Mon Sep 11, 2006 3:45 am    Post subject: Reply with quote

The overlay is a good thing, really ! :) So i use it now:
Code:
# eix ^portage$
[I] sys-apps/portage
     Available versions:  2.0.51.22-r3 2.0.51.22-r3[1] 2.0.54-r2 2.0.54-r2[1] 2.1-r2 2.1-r2[1] 2.1.1 2.1.1[1]
     Installed:           2.1.1
     Homepage:            http://www.gentoo.org/proj/en/portage/index.xml
     Description:         The Portage Package Management System. The primary package management and distribution system for Gentoo.

[1] /gentoo/overlay



I have tried to quickpkg a package and emerge it from the binary:
Code:
# quickpkg gftp
 * Building package for gftp-2.0.18-r4 ...                                                                                                 [ ok ]

 * Packages now in /gentoo/packages:
 * gftp-2.0.18-r4: 436K
# emerge -k gftp
Calculating dependencies... done!
Package list for slot = 0
        ['binary', '/', 'net-ftp/gftp-2.0.18-r4', 'merge']

>>> Emerging (1 of 1) net-ftp/gftp-2.0.18-r4 to /
>>> Extracting info
>>> Extracting gftp-2.0.18-r4
>>> Merging net-ftp/gftp-2.0.18-r4 to /
>>> Safely unmerging already-installed instance of net-ftp/gftp-2.0.18-r4 ...
>>> Regenerating /etc/ld.so.cache...
>>> Original instance of net-ftp/gftp-2.0.18-r4 unmerged safely.
>>> Regenerating /etc/ld.so.cache...
>>> net-ftp/gftp-2.0.18-r4 merged.

>>> No packages selected for removal by clean.

>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.


So that's work and try it on portage:
Code:
# quickpkg portage
 * Building package for portage-2.1.1 ...                                                                                                  [ ok ]

 * Packages now in /gentoo/packages:
 * portage-2.1.1: 368K
Zentoo-1 log # emerge -k portage
Calculating dependencies... done!
Package list for slot = 0
        ['binary', '/', 'sys-apps/portage-2.1.1', 'merge']

>>> Emerging (1 of 1) sys-apps/portage-2.1.1 to /
>>> Extracting info
>>> Extracting portage-2.1.1
tar: ./etc/portage/.keep_sys-apps_portage-0: Cannot open: Not a directory
tar: ./usr/lib/portage: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin/archive-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/chkcontents: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/clean_locks: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dispatch-conf: Cannot open: No such file or directory
tar: Skipping to next header
    -------- Several similar lines --------
tar: ./usr/lib/portage/pym/portage_util.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_versions.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/xpak.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: Error exit delayed from previous errors
!!! Error Extracting /gentoo/packages/All/portage-2.1.1.tbz2
tail: cannot open `/gentoo/log/sys-apps:portage-2.1.1:20060911-033418.log' for reading: No such file or directory
Package sys-apps/portage-2.1.1 failed to emerge
Please take a look at the file /gentoo/log/sys-apps:portage-2.1.1:20060911-033418.log


So that still doesn't work for portage :cry:


Anyway i should inspect it further the next days... At the moment i could easily come back to stable portage if i have some problem and use unstable one with the parallel patch from the overlay since portage does work.
_________________
Kernel 3.14.6| Gcc 4.8.3 | Glibc 2.19
Core i7 990x @ 4.0GHz | X58 | 12Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Mon Sep 11, 2006 5:07 am    Post subject: Reply with quote

what happens when you do this:
Code:

mkdir /tmp/portage-bin
cd /tmp/portage-bin
tar xjpf /usr/portage/packages/All/portage-2.1.1.tbz2
ls -R
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Mon Sep 11, 2006 5:55 am    Post subject: Reply with quote

ok, found the problem.
Code:

$ tar tvjpf /usr/portage/packages/All/portage-2.1.1.tbz2
...
...
lrwxrwxrwx root/root         0 2006-09-01 23:57 ./usr/lib -> lib64
drwxr-xr-x root/root         0 2005-12-28 20:42 ./usr/lib/portage/
drwxr-xr-x root/root         0 2006-09-10 15:31 ./usr/lib/portage/pym/
....

if usr/lib is a link to lib64 and lib64 is not created, usr/lib/portage/pym can't be created and niether can the files under it. Welcome to amd64!!

I will see if how it got that way. It looks like a portage bug.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Mon Sep 11, 2006 6:18 am    Post subject: Reply with quote

This bug should reproduce even with the unpatched portage. Can you please try with the original portage 2.1.1, then quickpkg portage and emerge -k portage. I have run out of installs to really try this... :(
Back to top
View user's profile Send private message
razze
Apprentice
Apprentice


Joined: 09 Mar 2005
Posts: 159
Location: Espoo, Finland

PostPosted: Mon Sep 11, 2006 6:51 am    Post subject: Reply with quote

I'm also on amd64 (mostly stable), and installed the complete version (not the patch!) of parallel-compile-enabled portage via an overlay. I have done some testings, and everythings seems to be working OK for me.
_________________
--------------------------
Linux User #393524

AMD Athlon 64 X2 3800+ @2,3 GHz, Asus A8V, VIA K8T800PRO, Nvidia GeForce 7600GS, 2 Gb Dual Channel DDR RAM, 2x Samsung SP120 200Gb, 2x WD 320Gb, Plextor PX-716A, linux-2.6.31-gentoo-r10
Back to top
View user's profile Send private message
Zentoo
Apprentice
Apprentice


Joined: 18 Nov 2002
Posts: 171
Location: /dev/console

PostPosted: Mon Sep 11, 2006 3:26 pm    Post subject: Bug in portage on ~amd64 Reply with quote

@devsk:

ok so i've test four combinaison of
Code:

 emerge portage
 quickpkg portage
 emerge -k portage


with your patched overlay:

Code:
portage-2.1-r2 => OK
portage-2.1.1 => FAILED


without patched, normal portage version:
Code:
portage-2.1-r2 => OK
portage-2.1.1 => FAILED


So it seems that's really a bug from the original 2.1.1 version specific to the link /lib on /lib64.




In case you need the bash answer to your previous request:

Code:
mkdir /tmp/portage-bin
cd /tmp/portage-bin
tar xjpf /usr/portage/packages/All/portage-2.1.1.tbz2
ls -R


Code:
portage-bin # tar xjpf /gentoo/packages/All/portage-2.1.1.tbz2
tar: ./etc/portage/.keep: Cannot open: Not a directory
tar: ./usr/lib/portage: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin: Cannot mkdir: No such file or directory
tar: ./usr/lib/portage/bin/archive-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/chkcontents: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/clean_locks: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dispatch-conf: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dobin: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/doconfd: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/bin/dodir: Cannot open: No such file or directory
tar: Skipping to next header
----->8----->8----->8----->8----->8----->8----->8---------
tar: ./usr/lib/portage/bin/donewins: Cannot create symlink to `newins': No such file or directory
tar: ./usr/lib/portage/bin/dosbin: Cannot open: No such file or directory
tar: Skipping to next header
----->8----->8----->8----->8----->8----->8----->8------------
tar: ./usr/lib/portage/pym/portage_update.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_util.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/portage_versions.py: Cannot open: No such file or directory
tar: Skipping to next header
tar: ./usr/lib/portage/pym/xpak.py: Cannot open: No such file or directory
tar: Skipping to next header

bzip2: (stdin): trailing garbage after EOF ignored
tar: Error exit delayed from previous errors


Code:
portage-bin # tree -natf
.
|-- ./etc
|   |-- ./etc/portage -> /gentoo/config
|   |-- ./etc/dispatch-conf.conf
|   |-- ./etc/env.d
|   |   `-- ./etc/env.d/05portage.envd
|   |-- ./etc/etc-update.conf
|   |-- ./etc/make.conf.example
|   `-- ./etc/make.globals
`-- ./usr
    |-- ./usr/lib -> lib64
    |-- ./usr/bin
    |   |-- ./usr/bin/ebuild -> ../lib/portage/bin/ebuild
    |   |-- ./usr/bin/emerge -> ../lib/portage/bin/emerge
    |   |-- ./usr/bin/portageq -> ../lib/portage/bin/portageq
    |   |-- ./usr/bin/repoman -> ../lib/portage/bin/repoman
    |   |-- ./usr/bin/tbz2tool -> ../lib/portage/bin/tbz2tool
    |   `-- ./usr/bin/xpak -> ../lib/portage/bin/xpak
    |-- ./usr/sbin
    |   |-- ./usr/sbin/archive-conf -> ../lib/portage/bin/archive-conf
    |   |-- ./usr/sbin/dispatch-conf -> ../lib/portage/bin/dispatch-conf
    |   |-- ./usr/sbin/emaint -> ../lib/portage/bin/emaint
    |   |-- ./usr/sbin/emerge-webrsync -> ../lib/portage/bin/emerge-webrsync
    |   |-- ./usr/sbin/env-update -> ../lib/portage/bin/env-update
    |   |-- ./usr/sbin/etc-update -> ../lib/portage/bin/etc-update
    |   |-- ./usr/sbin/fixpackages -> ../lib/portage/bin/fixpackages
    |   |-- ./usr/sbin/quickpkg -> ../lib/portage/bin/quickpkg
    |   `-- ./usr/sbin/regenworld -> ../lib/portage/bin/regenworld
    `-- ./usr/share
        |-- ./usr/share/doc
        |   `-- ./usr/share/doc/portage-2.1.1
        |       |-- ./usr/share/doc/portage-2.1.1/ChangeLog.gz
        |       |-- ./usr/share/doc/portage-2.1.1/NEWS.gz
        |       `-- ./usr/share/doc/portage-2.1.1/RELEASE-NOTES.gz
        `-- ./usr/share/man
            |-- ./usr/share/man/man1
            |   |-- ./usr/share/man/man1/dispatch-conf.1.gz
            |   |-- ./usr/share/man/man1/ebuild.1.gz
            |   |-- ./usr/share/man/man1/emaint.1.gz
            |   |-- ./usr/share/man/man1/emerge.1.gz
            |   |-- ./usr/share/man/man1/env-update.1.gz
            |   |-- ./usr/share/man/man1/etc-update.1.gz
            |   |-- ./usr/share/man/man1/quickpkg.1.gz
            |   `-- ./usr/share/man/man1/repoman.1.gz
            `-- ./usr/share/man/man5
                |-- ./usr/share/man/man5/ebuild.5.gz
                |-- ./usr/share/man/man5/make.conf.5.gz
                `-- ./usr/share/man/man5/portage.5.gz

12 directories, 35 files

(that's on the patched overlay version)



I have two question:

1) is the portage-2.1-r2 patched for parallel too ?
2) how do you handle finally the -jX stuff from make.conf in parallel mode ?

I have done a sucessfull system compilation this night about 114 pkgs in parallel mode in 123m43s
But i cannot check with my last system compile since an upgrade have add 10 pkgs to the system. (but just for infomation, it was 104 pkgs in 134m51s).
I should launch a new one without parallel tonight to compare and try to don't update my system since ... (gentoo addiction !) :wink:
_________________
Kernel 3.14.6| Gcc 4.8.3 | Glibc 2.19
Core i7 990x @ 4.0GHz | X58 | 12Gb
ACCEPT_KEYWORDS="~amd64"
CFLAGS="-march=native -O2 -pipe"
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2734
Location: Bay Area, CA

PostPosted: Mon Sep 11, 2006 3:59 pm    Post subject: Re: Bug in portage on ~amd64 Reply with quote

Zentoo wrote:

I have two question:

1) is the portage-2.1-r2 patched for parallel too ?

in the overlay that I provided, only 2.1.1 is patched with parallel. 2.1-r2 will need some work for applying the patch cleanly.
Zentoo wrote:

2) how do you handle finally the -jX stuff from make.conf in parallel mode ?

I don't do anything to -jX. So, if you have a dual core processor and have set MAKEOPTS=-j3, you will see that some of the times you might have a lot of gcc processes. But so far, I have noticed that dual cores combined with 2.6 kernel handle it well. It may not make any sense to reduce MAKEOPTS because certain large packages will delay the completion of a particular merge slot anyway.
Zentoo wrote:

I have done a sucessfull system compilation this night about 114 pkgs in parallel mode in 123m43s
But i cannot check with my last system compile since an upgrade have add 10 pkgs to the system. (but just for infomation, it was 104 pkgs in 134m51s).
I should launch a new one without parallel tonight to compare and try to don't update my system since ... (gentoo addiction !) :wink:

'emerge -e system' is probably not where it will shine the most (I saw about 15% improvement for system). Its 'emerge -uDN world' where the parallel will shine because of the diversity of the packages being emerged. So, the dividends will be paid long term...:) Try 'emerge -pev world' to see the slot arrangement and you will know the tree has lot of parallelism.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum