Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Why isn't Portage multithreaded?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
sparc
n00b
n00b


Joined: 18 Aug 2005
Posts: 67
Location: Stockholm, Sweden

PostPosted: Mon Feb 28, 2011 10:47 pm    Post subject: Why isn't Portage multithreaded? Reply with quote

Hi,

I have a very simple question to ask. Why aren't emerge, equery and the rest of the tools multithreaded?

I've searched the man pages and I haven't found an option to enable such a function. But let me elaborate more. I'm of course not referring to the --jobs functionality; performing a thorough emerge world with all the bells and wistles (-D --with-bdeps=y etc) has an initial overhead that accounts to multiple minutes. This becomes quite frustrating when one does not know what to expect and needs to verify the list of packages before. When this preview results in changes (mask, unmask and use flags) and the procedure has to be repeated, the whole task is just tedious. I have an i7, and I feel being significantly let down when gkrellm reports just one core at 99% while the rest of the system is idling.

As I can understand the process of dependency structuring, it has a big potential of parallelization; but to be honest I haven't bothered to go through the source code... yet! So here it is again. Why isn't this potential being exploited? Why isn't emerge and the rest of the tools multithreaded?
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Mon Feb 28, 2011 11:48 pm    Post subject: Reply with quote

On my 4 year old Core2 duo:

cat/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family   : 6
model      : 15
model name   : Intel(R) Core(TM)2 CPU         T7200  @ 2.00GHz
stepping   : 6
cpu MHz      : 1995.006
cache size   : 4096 KB
physical id   : 0
siblings   : 2
core id      : 0
cpu cores   : 2
apicid      : 0
initial apicid   : 0
fpu      : yes
fpu_exception   : yes
cpuid level   : 10
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow
bogomips   : 3990.01
clflush size   : 64
cache_alignment   : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family   : 6
model      : 15
model name   : Intel(R) Core(TM)2 CPU         T7200  @ 2.00GHz
stepping   : 6
cpu MHz      : 1995.006
cache size   : 4096 KB
physical id   : 0
siblings   : 2
core id      : 1
cpu cores   : 2
apicid      : 1
initial apicid   : 1
fpu      : yes
fpu_exception   : yes
cpuid level   : 10
wp      : yes
flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow
bogomips   : 3990.02
clflush size   : 64
cache_alignment   : 64
address sizes   : 36 bits physical, 48 bits virtual


time emerge -epv world:
Calculating dependencies            ... done!
real   0m30.096s
user   0m28.680s
sys   0m1.250s


Are you saying that it takes several minutes on your computer?
_________________
Who controls the past now, controls the future
Who controls the present now, controls the past.
Back to top
View user's profile Send private message
sparc
n00b
n00b


Joined: 18 Aug 2005
Posts: 67
Location: Stockholm, Sweden

PostPosted: Tue Mar 01, 2011 12:13 am    Post subject: Reply with quote

You should read more carefully... I am just saying that the following:
Code:

$ emerge -pvnuD --with-bdeps=y world

along with ~1000 lines worth of keyword and special use flag entries gives me the following on an 64-bit i7 with 4GB RAM & ext4 on a sataII 7200rpm disk
Code:

real    4m7.879s
user    4m3.919s
sys     0m2.752s


As you can assume from the ~1000 lines I mentioned, I keep a pretty un-stable system. The reason is a different story, but no it is not for fun. The bottom line is that whenever there are more than 10 packages to update, I have to manually go through the process of unmasking dependencies; this results in running emerge quite a few times. Multiply each by five minutes and you can understand why I consider a sequential emerge a tedious process.

By the way, while searching the man pages for a multithread option, I found --autounmask. How exactly does it work and where will the keyword entries be stored? I keep a non-basic file hierarchy and I do not want to risk messing it up.
Back to top
View user's profile Send private message
Naib
Advocate
Advocate


Joined: 21 May 2004
Posts: 4110
Location: Removed by Neddy

PostPosted: Tue Mar 01, 2011 12:25 am    Post subject: Reply with quote

such queries are more i/o bound then cpu bound. making it threaded wouldn't really provide any speed benefit if the same data has to be fetched from a slow turning hunk of plated glass
_________________
A free press is the unsleeping guardian of every other right that free men prize; it is the most dangerous foe of tyranny. Where men have the habit of liberty, the Press will continue to be the vigilant guardian of the rights of the ordinary citizen.
Back to top
View user's profile Send private message
sparc
n00b
n00b


Joined: 18 Aug 2005
Posts: 67
Location: Stockholm, Sweden

PostPosted: Tue Mar 01, 2011 12:54 am    Post subject: Reply with quote

Thanks for the input Naib. I thought the same at first also. However, even an idling firefox produces more file I/O than this query on average (according to iotop). Of course there are a few spikes than climb up to the MB per sec layer, but they last less than a dozen seconds all together On the contrary, I get one hardware thread at constant 99%. That alone is clue that the process is not constraint by the hdd; if it was it would have to wait for the data, thus lowering the CPU utilization.

In any case, I'm not here to complain. I want to know if there is a specific design decision behind this lack of functionality. If not however, I would really love to help add it.
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2526
Location: Richmond, BC, Canada

PostPosted: Tue Mar 01, 2011 1:23 am    Post subject: Reply with quote

No portage is not multi threaded. As naib has said, much of the data loaded by emerge is disk/file based. So will not see much benefit from multi threading. There is also a timing issue for portage that needs data loaded in a certain order for some things in order for the profile and USE flag stacking to work as expected. Another is that one of the portage backends, namely the sqlite one cannot be run from a thread. In porthole we run portage in threads for many things. and it works, but porthole users cannot use the portage sqlite backend.

One of the big problems with emerge/portage in general is the vast amount of code that has to load and initialize, as well as the data. It has been improving these last years, but it is a very large app. Porthole loads portage once, so searches and things tend to be much faster than from the command line. What you need is an interactive session where portage & data stays loaded and ready. I have been working on gentoolkit code for awhile now. I got involved in order to make usable api's for many of its modules so that I can make a plug-in for it in porthole. Since equery loads and gathers it's data from portage, it too gets the slowdown by initializing and gathering data for each run of the command line. When I get the porthole plug-in done. It won't have that same slowdown since it will only load and initialize once at the start.

So for now, to get and see dependency graphs for pkgs, I would recommend you try porthole. It has a fully dynamic dependency view that is capable of following unsatisfied dependencies to the ultimate end. And it is quick too. I am currently working on layman code now, since I've taken over maintainership/development. I am adding a new api so other apps can use and run it. I have also started to create a plug-in for porthole. Once that is done it will be easier to create a gentoolkit plug-in with equery searches as well.

Oh and I think you'll find porthole can generate a complete upgrade list of every package on your system, not just pvnuD list in less than 20 seconds. Even when I haven't updated in a long while, it never takes more than 15 sec. typically about 3-5 sec. The main reason is that it doesn't build dependency graphs, it has a list of all installed pkgs, then checks the entire list for available upgrades. Then you can decide what you want to upgrade, what deps are needed for them, etc. Emerge will look after needed deps at emerge time. When I get more time, I'll add more portage api to calculate dependency graphs from portage. But that is a different project.

Another option is to try pkgcore. It was designed from the start for good code structure and speed and to be a next gen. portage. It does not have as many features as portage, but will run circles around it speed wise. A dependency graph like the one you posted, the times would be less than a minute, probably less than 30sec from my experience with using it.
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Tue Mar 01, 2011 1:56 am    Post subject: Reply with quote

keywords, use and even overlays shouldnt make that big of an impact, what do you get with:

hdparm:
hdparm -tT /dev/root


With your specs it should be faster than mine:

hdparm results:
 Timing cached reads:   5120 MB in  2.00 seconds = 2564.65 MB/sec
 Timing buffered disk reads:  146 MB in  3.03 seconds =  48.18 MB/sec


btw I have:

find /etc/portage/ -type f:
/etc/portage/savedconfig/sys-apps/busybox-1.17.4
/etc/portage/postsync.d/q-reinitialize
/etc/portage/.keep_sys-apps_portage-0
/etc/portage/package.keywords/x11-themes/gdm-themes
/etc/portage/package.keywords/x11-themes/emerald-themes
/etc/portage/package.keywords/media-libs/alsa-lib
/etc/portage/package.keywords/dev-python/compizconfig-python
/etc/portage/package.keywords/x11-plugins/compiz-plugins-main
/etc/portage/package.keywords/x11-plugins/compiz-plugins-unsupported
/etc/portage/package.keywords/x11-plugins/compiz-plugins-extra
/etc/portage/package.keywords/net-irc/irssi
/etc/portage/package.keywords/app-emulation/virtualbox-additions
/etc/portage/package.keywords/app-emulation/virtualbox
/etc/portage/package.keywords/app-emulation/virtualbox-modules
/etc/portage/package.keywords/sys-process/procps
/etc/portage/package.keywords/x11-libs/compiz-bcop
/etc/portage/package.keywords/x11-libs/libcompizconfig
/etc/portage/package.keywords/x11-libs/compizconfig-backend-gconf
/etc/portage/package.keywords/media-sound/alsa-utils
/etc/portage/package.keywords/media-sound/alsa-headers
/etc/portage/package.keywords/sys-kernel/genkernel
/etc/portage/package.keywords/sys-kernel/hardened-sources
/etc/portage/package.keywords/net-analyzer/nmap
/etc/portage/package.keywords/net-analyzer/jnettop
/etc/portage/package.keywords/x11-wm/emerald
/etc/portage/package.keywords/x11-wm/compiz
/etc/portage/package.keywords/x11-wm/compiz-fusion
/etc/portage/package.keywords/sys-apps/portage
/etc/portage/package.keywords/sys-boot/grub
/etc/portage/package.keywords/www-plugins/adobe-flash
/etc/portage/package.keywords/x11-apps/ccsm
/etc/portage/package.unmask/x11-drivers/nvidia-drivers
/etc/portage/package.unmask/media-video/nvidia-settings
/etc/portage/package.unmask/sys-apps/portage
/etc/portage/bin/post_sync
/etc/portage/profile/use.mask
/etc/portage/package.use/media-libs/lcms
/etc/portage/package.use/media-libs/jpeg
/etc/portage/package.use/media-libs/libpng
/etc/portage/package.use/dev-db/sqlite
/etc/portage/package.use/sys-block/parted
/etc/portage/package.use/net-wireless/wpa_supplicant
/etc/portage/package.use/dev-lang/python
/etc/portage/package.use/sys-auth/consolekit
/etc/portage/package.use/sys-boot/grub
/etc/portage/package.use/app-text/poppler
/etc/portage/package.use/dev-libs/libgpg-error
/etc/portage/package.use/dev-libs/libgcrypt
/etc/portage/package.use/sys-fs/udev


find /usr/local/portage/ -type f | grep ebuild:
/usr/local/portage/sys-kernel/hardened-sources/hardened-sources-2.6.37-r2.ebuild
/usr/local/portage/sys-kernel/hardened-sources/hardened-sources-2.6.37-r3.ebuild
/usr/local/portage/sys-boot/boot-update/boot-update-1.4.12.ebuild
/usr/local/portage/sys-boot/boot-update/boot-update-1.4.11.ebuild
/usr/local/portage/sys-boot/boot-update/boot-update-1.5.2.ebuild
/usr/local/portage/sys-boot/grub/grub-1.98-r2.ebuild
/usr/local/portage/sys-boot/grub/grub-1.98.ebuild


portageq envvar USE:
X a52 aac aalib acl acpi adns aio alisp alsa alsa_cards_hda-intel alsa_pcm_plugins_adpcm alsa_pcm_plugins_alaw alsa_pcm_plugins_asym alsa_pcm_plugins_copy alsa_pcm_plugins_dmix alsa_pcm_plugins_dshare alsa_pcm_plugins_dsnoop alsa_pcm_plugins_empty alsa_pcm_plugins_extplug alsa_pcm_plugins_file alsa_pcm_plugins_hooks alsa_pcm_plugins_iec958 alsa_pcm_plugins_ioplug alsa_pcm_plugins_ladspa alsa_pcm_plugins_lfloat alsa_pcm_plugins_linear alsa_pcm_plugins_meter alsa_pcm_plugins_mmap_emul alsa_pcm_plugins_mulaw alsa_pcm_plugins_multi alsa_pcm_plugins_null alsa_pcm_plugins_plug alsa_pcm_plugins_rate alsa_pcm_plugins_route alsa_pcm_plugins_share alsa_pcm_plugins_shm alsa_pcm_plugins_softvol amd64 amr apache2_modules_actions apache2_modules_alias apache2_modules_auth_basic apache2_modules_authn_alias apache2_modules_authn_anon apache2_modules_authn_dbm apache2_modules_authn_default apache2_modules_authn_file apache2_modules_authz_dbm apache2_modules_authz_default apache2_modules_authz_groupfile apache2_modules_authz_host apache2_modules_authz_owner apache2_modules_authz_user apache2_modules_autoindex apache2_modules_cache apache2_modules_cgi apache2_modules_cgid apache2_modules_dav apache2_modules_dav_fs apache2_modules_dav_lock apache2_modules_deflate apache2_modules_dir apache2_modules_disk_cache apache2_modules_env apache2_modules_expires apache2_modules_ext_filter apache2_modules_file_cache apache2_modules_filter apache2_modules_headers apache2_modules_include apache2_modules_info apache2_modules_log_config apache2_modules_logio apache2_modules_mem_cache apache2_modules_mime apache2_modules_mime_magic apache2_modules_negotiation apache2_modules_rewrite apache2_modules_setenvif apache2_modules_speling apache2_modules_status apache2_modules_unique_id apache2_modules_userdir apache2_modules_usertrack apache2_modules_vhost_alias audacious audit bash-completion bashlogger berkdb bolddiag bzip2 cairo cameras_ptp2 caps cddb cdr cleartype cli collectd_plugins_df collectd_plugins_interface collectd_plugins_irq collectd_plugins_load collectd_plugins_memory collectd_plugins_rrdtool collectd_plugins_swap collectd_plugins_syslog consolekit cracklib crypt cscope cups cursors custom-cflags custom-optimization cxx dbus dell dga dirac disk-partition djbfft djvu dri dvd dvdr dvi elibc_glibc emerald encode faac fam fat fbcon fbcondecor ffmpeg fftw flac fontconfig fpx frei0r fuse gallium gcdmaster gd gdbm gdm gdu gif glade gmp gnome gnutls gphoto2 gpm gpsd_protocols_aivdm gpsd_protocols_ashtech gpsd_protocols_earthmate gpsd_protocols_evermore gpsd_protocols_fv18 gpsd_protocols_garmin gpsd_protocols_garmintxt gpsd_protocols_gpsclock gpsd_protocols_itrax gpsd_protocols_mtk3301 gpsd_protocols_navcom gpsd_protocols_nmea gpsd_protocols_ntrip gpsd_protocols_oceanserver gpsd_protocols_oldstyle gpsd_protocols_oncore gpsd_protocols_rtcm104v2 gpsd_protocols_rtcm104v3 gpsd_protocols_sirf gpsd_protocols_superstar2 gpsd_protocols_timing gpsd_protocols_tnt gpsd_protocols_tripmate gpsd_protocols_tsip gpsd_protocols_ubx graphviz gs gsm gstreamer gtk hal hardened hdri icons iconv idn ieee1394 input_devices_evdev input_devices_keyboard input_devices_mouse input_devices_virtualbox iphone ipv6 ithreads java jpeg jpeg2k justify kdrive kerberos kernel_linux lame lcd_devices_bayrad lcd_devices_cfontz lcd_devices_cfontz633 lcd_devices_glk lcd_devices_hd44780 lcd_devices_lb216 lcd_devices_lcdm001 lcd_devices_mtxorb lcd_devices_ncurses lcd_devices_text lcms ldap libcaca libffi libnl libnotify linguas_en linguas_en_US llvm llvm-gcc lqr lzma lzo mad mmap mmx mng modules motif mp3 mpeg mudflap multilib multiuser nautilus ncurses network-cron nls nptl nptlonly nsplugin ntp nvidia ogg openexr opengl openmp optimization optimized-qmake pam pcre perl php_targets_php5-3 pic png policykit pppd python python3 raster raw readline rtmp ruby_targets_ruby18 schroedinger sdl session slp speex sse sse2 sse3 sse4 ssl ssse3 startup-notification strong-optimization svg sysfs tcpd theora threads tiff truetype udis86 unicode unsupported urandom userland_GNU v4l v4l2 video_cards_nv video_cards_nvidia vim vim-syntax virtualbox vorbis vpx wmf xcb xml xmp xorg xpm xtables_addons_account xtables_addons_chaos xtables_addons_condition xtables_addons_delude xtables_addons_dhcpmac xtables_addons_fuzzy xtables_addons_geoip xtables_addons_iface xtables_addons_ipmark xtables_addons_ipp2p xtables_addons_ipset xtables_addons_ipv4options xtables_addons_length2 xtables_addons_logmark xtables_addons_lscan xtables_addons_pknock xtables_addons_psd xtables_addons_quota2 xtables_addons_rawnat xtables_addons_steal xtables_addons_sysrq xtables_addons_tarpit xtables_addons_tee xv xvid zlib


Even with that I get:

time emerge -uDpvN --with-bdeps=y world:
These are the packages that would be merged, in order:

Calculating dependencies           ... done!                 

Total: 0 packages, Size of downloads: 0 kB

real   0m18.877s
user   0m17.630s
sys   0m0.700s


I think the problem is in your kernel, not portage.
_________________
Who controls the past now, controls the future
Who controls the present now, controls the past.
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2173
Location: $TERM

PostPosted: Tue Mar 01, 2011 2:26 am    Post subject: Reply with quote

That's why multi core processors always suck. The coding is difficult, and many times impossible.

I think it all happened cause of Microsoft's rubbish CPU scheduler (worst in the world); to compensate for it the manufactures (like always) changed their design.

BTW when it comes to disk I/O intensive operations, multi threading is bound to give issues unless the task is waaaay too much CPU extensive, the HDD head will rocket here and there further reducing the data transfer.
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Tue Mar 01, 2011 2:43 am    Post subject: Reply with quote

As far as portage is concerned it is probably mainly because threads is an optional use flag in python, so the devs would have to put in additional code if python isn't threaded.

Make and alot of other programs seem to do very well in multiple core processors.

What really sucks is when you have a 24 core server and an app that only uses one core.
_________________
Who controls the past now, controls the future
Who controls the present now, controls the past.
Back to top
View user's profile Send private message
The Doctor
Veteran
Veteran


Joined: 27 Jul 2010
Posts: 1443

PostPosted: Tue Mar 01, 2011 2:50 am    Post subject: Reply with quote

dE_logics wrote:
The coding is difficult, and many times impossible.

BTW when it comes to disk I/O intensive operations, multi threading is bound to give issues unless the task is waaaay too much CPU extensive, the HDD head will rocket here and there further reducing the data transfer.


As a student who looked way into multi-threaded applications, I would like to point a little deeper. Mulit-threaded applications are only shine when the number of threads equals or is less than the processor number. This is the hard part to code. You need to program for x number of threads. The data you want to process then needs to be in ram, because the hard drive can only retrieve one file at a time.

If you want something that will take advantage of your i7, then you need an app that loads a ton of data, and then processes that data independently. I used threads in an attempted algebraic calculator to write number objects in a standard form. This worked, because a number object does not depend on how others are processed and a queue could hold the unprocessed data. For something like portage, there is really no point, because a, portage needs to be simple. multi-threaded is necessarily complicated. B, when you run portage, the real time is spent compiling. And finally c, there is nothing to run parallel and some portage has some applications that can't run in a thread.
_________________
First things first, but not necessarily in that order.
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1374
Location: Bardowick, Germany

PostPosted: Tue Mar 01, 2011 9:04 am    Post subject: Reply with quote

Maybe you might want to switch your portage to use sqlite instead of flat files?
Code:
 # time emerge -pvnuD --with-bdeps=y world
(... snip ...)
real    0m36.063s
user    0m30.688s
sys     0m0.780s
My environment:
Code:
# wc -l /etc/portage/package.*
    0 /etc/portage/package.accept_keywords
  127 /etc/portage/package.env
  424 /etc/portage/package.keywords
    3 /etc/portage/package.license
    0 /etc/portage/package.mask
    0 /etc/portage/package.provided
    3 /etc/portage/package.unmask
  127 /etc/portage/package.use
  684 total
Not as much as in yours, but my dual-core i7 is currently running two virtual machines (One Windows XP with a VPN connection and a gentoo box that is currently compiling a new kernel) and I do not think that 36 seconds is too much.
_________________
I *do* know that I easily aggravate people due to my condensed writing. Rule of thumb: If I wrote anything that can be understood in two different ways, and one way offends you, then I meant the other! ;)
Back to top
View user's profile Send private message
aCOSwt
Moderator
Moderator


Joined: 19 Oct 2007
Posts: 2537
Location: Hilbert space

PostPosted: Tue Mar 01, 2011 9:28 am    Post subject: Reply with quote

penguin swordmaster wrote:
Mulit-threaded applications are only shine when the number of threads equals or is less than the processor number. This is the hard part to code. You need to program for x number of threads.

Multi-threaded systems are only shine when the number of kernel-threads is less or equal than the number of processors.
dE_logics is then right pointing out the scheduler's responsibility because the efficiency will greatly depend on the implemented scheduling model.

User threads are not truly scheduleable entities. And the kernel has no knowledge of them.
The kernel simply schedules the underlying-process or LWP, [i.e kernel-thread] which in turn uses library functions to schedule the user-threads that are associated with it.
When the LWP is preempted, so are all its user-threads.

Likewise, if a user-thread makes a blocking system call, it blocks the underlying LWP. If the user-process has only one LWP, all its user-threads are blocked.
=> In this case a multi-threaded system has got the same efficiency than a single-threaded one.

Then, if the scheduler implements :

- A [N:1] scheduling model (the N user-threads of a user-process associated to a single kernel-thread) No application will benefits from being multithreaded.
- A [N:M] scheduling model (whatever N user-threads associated to M kernel-threads) (SCHED_4BSD as an efficient example), penguin_swordmaster is right. No process heavily relying on blocking IO calls will gain much from having more threads than the number of processors.
- A [1:1] scheduling model (A single user-thread associated to a single kernel-thread) (SCHED_ULE as a good example), any user-thread making a blocking IO call will only preempt itself (In a GIANT LOCK free world of course) this leading to heavy gains for an n-threaded application irrespective of the number of processors.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4067

PostPosted: Tue Mar 01, 2011 8:24 pm    Post subject: Reply with quote

Keeping your portage tree in squashfs+aufs gives an enormous boost, since most of the IO-bound vanishes. I think it should decrease the 5 minutes to 30 seconds.
The 1000 lines config-files shouldn't be an issue - I have these also. (Yes, my cpu is much slower than yours).
Another hint to avoid frequent calling of emerge in your case is to use the new --autounmask option.
Back to top
View user's profile Send private message
sparc
n00b
n00b


Joined: 18 Aug 2005
Posts: 67
Location: Stockholm, Sweden

PostPosted: Tue Mar 01, 2011 9:54 pm    Post subject: Reply with quote

mv wrote:
Keeping your portage tree in squashfs+aufs gives an enormous boost, since most of the IO-bound vanishes. I think it should decrease the 5 minutes to 30 seconds.
The 1000 lines config-files shouldn't be an issue - I have these also. (Yes, my cpu is much slower than yours).
Another hint to avoid frequent calling of emerge in your case is to use the new --autounmask option.


First of all this operation is NOT I/O bound on any modern machine. Use simple logic to figure it out. If you do not trust logic (!?!) use iotop.

Second, in my second post I also ask how exactly the --autounmask option works, as the man page is far from descriptive.

Also let me clarify that I got the 5 minute duration with ~300 packages to be updated (kde 4.6.0 plus quite a bit more). Posting measurements of emerge taking seconds with 0 packages to update or the --empty-tree option is useless as it has nothing to do with my situation. Installing software the size of kde, while marked unstable, is meant to require unmasking a ton of dependencies (not included in the keyword files provided for KDE). I hope now you understand my scenario.

Finally, I was expecting a serious discussion with one of the developers as I came on this forum prepared to contribute. I do know parallel software, it has been my research topic for the past few years (PhD). I do not know portage internals however and as others mentioned it is too big to just read the source, especially with my very limited free time.
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Wed Mar 02, 2011 12:11 am    Post subject: Reply with quote

Is this a better comparison?

time emerge -uDpvN world (amd64):
These are the packages that would be merged, in order:

Calculating dependencies                   ... done!

Total: 0 packages, Size of downloads: 0 kB

real   0m21.632s
user   0m20.890s
sys   0m0.710s



time emerge -uDpvN world (~amd64):
These are the packages that would be merged, in order:

Calculating dependencies                   ... done!

Total: 347 packages (239 upgrades, 46 new, 2 in new slots, 60
reinstalls), Size of downloads: 548,822 kB

real    0m28.177s
user    0m24.270s
sys     0m0.730s


I'll trust strace to know that emerge is a 99% read operation, blind faith in a single tool like iotop will fsck you every time.
_________________
Who controls the past now, controls the future
Who controls the present now, controls the past.
Back to top
View user's profile Send private message
sparc
n00b
n00b


Joined: 18 Aug 2005
Posts: 67
Location: Stockholm, Sweden

PostPosted: Wed Mar 02, 2011 1:22 am    Post subject: Reply with quote

Here is one test you asked previously:
Code:

# hdparm -tT /dev/root

/dev/root:
 Timing cached reads:   14410 MB in  2.00 seconds = 7212.16 MB/sec
 Timing buffered disk reads:  290 MB in  3.01 seconds =  96.38 MB/sec

I hope this proves two tings: that it's not the file IO which causes the problem, and that your theory is wrong.

Next, about your emerge operation... Do not be so stubborn to compare your machine to mine. You know nothing about my installation. You know nothing about why I have an unstable system. Nonetheless you insist that all systems should behave just like yours.

I have 1826 packages installed. If I'm not mistaken, a thorough --with-bdeps emerge checks every to-be-updated/installed-package for missing dependencies and for conflicts against what is already installed. So it is safe to assume that the execution time is a direct consequence of the number of packages installed.

And even if all of the above are just crap. I have a Gentoo system that behaves as such. I saw a way to improve it and OFFERED to implement it MYSELF. What is so wrong with that even if you don't like my idea?

PS: a single hard disk can not feed a CPU with so much data as to keep it bound at 99% for 5 minutes. Since anything is slower than a hardware thread, any external bottleneck would reduce CPU utilization instantly.
Back to top
View user's profile Send private message
dol-sen
Developer
Developer


Joined: 30 Jun 2002
Posts: 2526
Location: Richmond, BC, Canada

PostPosted: Wed Mar 02, 2011 2:50 am    Post subject: Reply with quote

Ok, to clarify, my experience a little more with threading portage. Most of it is thread safe, with one notable exception the sqlite backend which must run from the main thread only. In porthole the imported portage code is used for data gathering only, so there is no concern for configs and things getting out of sync between threads. That could very well be a different story for a fully utilized emerge process.

For getting more advice as to where to start, the perils and things to watch out for in portage code. Get on irc and join #gentoo-portage on the freenode network. You'll want to talk to zmedico
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2333
Location: UK

PostPosted: Wed Mar 02, 2011 3:23 am    Post subject: Reply with quote

sparc wrote:
I saw a way to improve it and OFFERED to implement it MYSELF.

Sure thing, go for it. The source is right there, I'm sure at least one of the three major package managers will accept patches for multithreading.
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Wed Mar 02, 2011 3:44 am    Post subject: Reply with quote

I don't approach any problem with one solution in mind, IO was just my first guess as I have seen people unknowingly using a generic hd driver, disable swap, mmap, or other kernel options.

Your problem could be caused by many different reasons, inconsistent symbols caused by changed USE flags, emerging glibc or kernel headers and not running emerge -e world, running ricersfs, or CFLAGS like -O3 --omg-optimized I was just trying to help you look outside of the box.

The amount of packages you have isnt as drastic a number as you think. I have had systems with >2000 packages on about half the hardware using much slower versions of portage and even then it didn't take 5 minutes to resolve dependencies.

I don't have a problem with threading portage, but the fact is that portage in it's current state should not be anywhere near that slow. I see engineers and programmers all the time try to compensate for misconfigurations or crap code by throwing more ram and processor cores at it instead of fixing_the_problem.

If that is the road you want to walk down be my guest.
_________________
Who controls the past now, controls the future
Who controls the present now, controls the past.
Back to top
View user's profile Send private message
Yamakuzure
Veteran
Veteran


Joined: 21 Jun 2006
Posts: 1374
Location: Bardowick, Germany

PostPosted: Wed Mar 02, 2011 11:21 am    Post subject: Reply with quote

Okay, I give it a try:

Dependencies in 5 minutes:

There seems to be something wrong with your machine or your setup. Even with the number of your packages:
Code:
# LC_ALL="C"; time ACCEPT_KEYWORDS="~amd64" emerge --pretend --emptytree --update --newuse --with-bdeps y --update @world

These are the packages that would be merged, in order:

Calculating dependencies... done!
(... looooooong list ...)

Total: 1521 packages (331 upgrades, 42 new, 5 in new slots, 1143 reinstalls), Size of downloads: 454,815 kB
Fetch Restriction: 2 packages
Conflict: 1 block
Portage tree and overlays:
 [0] /usr/portage
 [1] /usr/local/portage
 [2] /var/lib/layman/vmware
 [3] /var/lib/layman/gamerlay
 [4] /var/lib/layman/sunrise

real    0m46.112s
user    0m33.404s
sys     0m0.817s
Look here, I use quite some overlays, and my local overlay has quite a few ebuilds in it (58 to be exact).
If I leave the --emptytree option out, I get 63 updates and the dependencies are calculated in 26 seconds.

On a second machine with an old Xeon-1 (32bit, Single-core) CPU and 1,5GiB RAM, the "non-emtpy-tree" version takes 2m14 for the calculation of 321 upgrades (1086 packages installed). So no, neither portage or our suggestions are the culprit, it's somewhere in your setup (we know nothing detailed about).

The test of the harddrive:
Code:
# hdparm -tT `mount | grep /usr | cut -d ' ' -f1`

/dev/sda5:
 Timing cached reads:   9772 MB in  2.00 seconds = 4887.67 MB/sec
 Timing buffered disk reads:  274 MB in  3.02 seconds =  90.70 MB/sec
So it is somewhat slower than yours.

--autounmask:
sparc wrote:
I have to manually go through the process of unmasking dependencies
No, you haven't.
If you want to install something like KDE-4.6.0 with so many new packages to unmask, I'd suggest pumping them into your package.keywords by a little script before using emerge --autounmask or app-portage/autounmask. After everything is installed, you can clean up after yourself with analyse from app-portage/gentoolkit or app-portage/portpeek automatically.
In fact before I installed KDE-4.6.0 I used this one-liner:
Code:
# for pkg in $(eix --only-names -C kde-*) ; do echo "=${pkg}-4.6.0::gentoo" >> /etc/portage/package.keywords ; done
and then app-portage/autounmask on kde-meta. (didn't know about the --autounmask option yet)
sparc wrote:
Second, in my second post I also ask how exactly the --autounmask option works, as the man page is far from descriptive.
What is the problem with the documentation?
man emerge wrote:
--autounmask [ y | n ]
Automatically unmask packages. If any configuration changes are required, then they will be displayed after the merge list and emerge will immediately abort. If the displayed configuration changes are sat‐
isfactory, you should copy and paste them into the specified configuration file(s). Currently, this only works for unstable KEYWORDS masks, LICENSE masks, and package.use settings.
I know this is pretty subjective, but for me this is crystal clear. Get a list what has to be changed and copy/paste it into the appropriate file.

Last but not least: Tone:
If you ask something but provide so little information that you feel yourself forced to say something like "You know nothing about..." then it is your fault, because you didn't give proper information.
As for your assumption what is safe to assume, yes, the number of packages is relevant, but if far inferior machines with far more packages than yours need a lot less time, then it is something about your setup or your machine that is wrong. Or to say it with your own words: "Use simple logic to figure it out."
_________________
I *do* know that I easily aggravate people due to my condensed writing. Rule of thumb: If I wrote anything that can be understood in two different ways, and one way offends you, then I meant the other! ;)
Back to top
View user's profile Send private message
sera
Developer
Developer


Joined: 29 Feb 2008
Posts: 991
Location: CET

PostPosted: Wed Mar 02, 2011 4:24 pm    Post subject: Reply with quote

For me the issue sounds more like portage aborting at the first issue than an issue with multi threading. Imagine portage would tell you all changes needed in a single pass.
Sure there are potential problems but no more than with autoumask.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4067

PostPosted: Wed Mar 02, 2011 5:58 pm    Post subject: Reply with quote

sparc wrote:
First of all this operation is NOT I/O bound on any modern machine. Use simple logic to figure it out. If you do not trust logic (!?!) use iotop.

I am wondering how you can make such a ridiculous claim without having it tried: Just measure the squashfs+aufs approach vs. the filesystem approach, and you will see
(my estimate 5min->30sec is based on experience).
This is also very logical: Your portage tree is scattered over many blocks on disks which need to be read in random order which increases search time enormously. Your processor simply has to wait until the corresponding block is reached, so I guess iotop will not give you useful information concerning this. In contrast, with the squashfs+aufs approach, the file containing the portage tree will be very quickly in RAM cache.
When you use SSD, things might be different - I have no experience with that. Also, I have much less RAM than you which might also change things. One really has to try before one can say something.
Quote:
in my second post I also ask how exactly the --autounmask option works, as the man page is far from descriptive.

I did not read all posts carefully. However, in case you are afraid: The --autounmask option of portage won't change your configuration files, it just gives more information than usual (and therefore portage can need slightly longer than usual - I guess this is why this is not the default but only an option). So just try and see.
Yamakuzure wrote:
In fact before I installed KDE-4.6.0 I used this one-liner:
Code:
# for pkg in $(eix --only-names -C kde-*) ; do echo "=${pkg}-4.6.0::gentoo" >> /etc/portage/package.keywords ; done

With modern portage you can also do something like
Code:
echo "kde-base/*:4.6" >> /etc/portage/package.keywords
Back to top
View user's profile Send private message
sparc
n00b
n00b


Joined: 18 Aug 2005
Posts: 67
Location: Stockholm, Sweden

PostPosted: Wed Mar 02, 2011 7:32 pm    Post subject: Reply with quote

On my faulty machine I get the following:
Code:

# time emerge --pretend --update --with-bdeps=y --update @world

These are the packages that would be merged, in order:

Calculating dependencies... done!
...
(package list)
...

real    0m36.832s
user    0m36.301s
sys     0m0.461s


Confused? Portage is a complex tool with too many options. The thing is that these options do not behave the same when combined together. See if I enter --emptytree above I get even shorter times (around 20 something seconds). The time changes if I enter --newuse also. So what is the problem, you might ask.

The problem is the following:
Code:

# time emerge --pretend --update --deep --with-bdeps=y --update @world

These are the packages that would be merged, in order:

Calculating dependencies... done!
...
(a slightly longer package list)
...

real    3m31.638s
user    3m28.788s
sys     0m2.550s


What changed? A simple option called --deep, in combination with everything else. Whatever the case on YOUR system, I'm not going to continue following this thread. That is because in my book the 30 seconds that satisfy you all is still slow performance. And as a last remark, if you all exit your comfort zone and start playing around with portage's options, you too can make it run for 3-5 minutes.

cheers
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4067

PostPosted: Wed Mar 02, 2011 8:40 pm    Post subject: Reply with quote

sparc wrote:
Whatever the case on YOUR system, I'm not going to continue following this thread.

Great introduction of you: Bashing around without understanding what you are really talking about, ignoring explanations and then even the further answers. If there were a kill-file for this forum you were now in mine.
Quote:
That is because in my book the 30 seconds that satisfy you all is still slow performance.

As I said you have no idea what you are talking about:
Code:
$ time sh -c 'echo /var/db/pkg/*/*/* /usr/portage/metadata/cache/*/* | xargs cat >/dev/null'
sh -c   0.54s user 28.86s system 85% cpu 34.500 total

And this is with squashfs. So you see where the 30 seconds come from.
Quote:
And as a last remark, if you all exit your comfort zone and start playing around with portage's options, you too can make it run for 3-5 minutes.

Do you really believe long-yeared gentoo users have not already used dozen's of portage's options? Your "slow" command (without --deep) takes about 30 seconds on the same machine which is much slower than yours: amd XP1700+ (32 bit only) with only 256 MB memory. With option --deep, this is in my experience the slowest portage call on this machine: It takes about 1:45 minutes. The reason here is rather clearly the lack of memory so that during backtracking probably the squashed file must be re-read several times. On a faster machine with more memory, both commands take about the 30 seconds which corresponds roughly to the time to read the data.
Back to top
View user's profile Send private message
boerKrelis
Apprentice
Apprentice


Joined: 01 Jul 2003
Posts: 237
Location: The Netherlands

PostPosted: Wed Mar 02, 2011 11:22 pm    Post subject: Reply with quote

sparc wrote:
First of all this operation is NOT I/O bound on any modern machine. Use simple logic to figure it out.

Well, it depends.
With a hot filecache:
Code:
emerge world -vuDp  15.57s user 0.27s system 99% cpu 15.898 total

Cold cache:
Code:
emerge world -vuDp  15.64s user 0.47s system 23% cpu 1:09.68 total


See? With a hot cache it's CPU bound, with a cold cache it isn't.

sparc wrote:
If you do not trust logic (!?!) use iotop.

IOtop does not show IOPS. Can I make it show them? On rotating media and with a usage pattern like with portage's, you want to be looking at IOPS.
I can make your disk come to a screeching halt (figuratively) without you seeing anything interesting in iotop.

Don't bother with the logic just yet. These are complex systems. Whip out a profiler. Then it gets interesting.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum