View previous topic :: View next topic |
Author |
Message |
vladimir_ttt n00b
Joined: 22 Feb 2014 Posts: 2
|
Posted: Sat Feb 22, 2014 8:46 am Post subject: Emerge is very slow |
|
|
Hi,
I've started to use gentoo around 11 years ago.
I had Celeron 266MHz processor with 64Mb memory.
Any "emerge" command has run on ot in at most 20-40 sec up to result (without compilation phase).
Now I have computer 40 times faster with 8Gb memory and...
Code: | time emerge -pv world |
First of all about times:
Code: |
real 6m26.305s
user 5m22.163s
sys 0m1.728s
|
And second if it is a system that hasn't be updated an year I recieve a lot of output like this
Code: |
dev-qt/qtcore:4
(dev-qt/qtcore-4.8.5-r1::gentoo, ebuild scheduled for merge) pulled in by
>=dev-qt/qtcore-4.8.5:4 required by (app-office/akonadi-server-1.10.3::gentoo, ebuild scheduled for merge)
~dev-qt/qtcore-4.8.5[aqua=,debug=] required by (dev-qt/qtsvg-4.8.5::gentoo, ebuild scheduled for merge)
(and 12 more with the same problems)
(dev-qt/qtcore-4.8.4-r4::gentoo, installed) pulled in by
~dev-qt/qtcore-4.8.4[-aqua,-debug,qt3support] required by (dev-qt/qt3support-4.8.4::gentoo, installed)
(and 11 more with the same problem)
dev-qt/qtdbus:4
(dev-qt/qtdbus-4.8.4::gentoo, installed) pulled in by
~dev-qt/qtdb
......
|
Who broke my perfect gentoo?
Why should I wait more than 6 min to receive no result?
I am almost on the edge to remove all my gentoos and pass to ubuntu!!! |
|
Back to top |
|
|
i92guboj Bodhisattva
Joined: 30 Nov 2004 Posts: 10315 Location: Córdoba (Spain)
|
Posted: Sat Feb 22, 2014 9:01 am Post subject: |
|
|
I guess the answer for both questions is that portage does not a lot more than it used to do ten years ago, specially when it comes to resolving dependencies and blockers.
Times are much shorter, and updates easier to handle, if you keep your system up to date. Otherwise, you have to deal with heavy changes that will no doubt require human intervention to be ironed out.
If you manage to refrain your Ubuntu rush and want to update your machines, I advice you to update sys-apps/portage the first. After that, emerge -auDvN world and paste here the blockers or whatever errors you are getting.
If you still get qt blockers, most of them can be solved by just forcing the update without taking deps into consideration. Then go from there... Something in this line might get you started:
Code: | $ qlist -I dev-qt --nocolor | while read package; do emerge --oneshot --nodeps $package; done |
|
|
Back to top |
|
|
TomWij Retired Dev
Joined: 04 Jul 2012 Posts: 1553
|
Posted: Sun Feb 23, 2014 1:39 am Post subject: |
|
|
Too much packages, too much overlays, too much unnecessary USE flags, ...; there are various reasons, maybe you are just in a situation that involves a lot of backtracking. Passing --backtrack=0 is as fast as it used to be (30 seconds to me); however, if you do that you need to have quite a skill to resolve things manually (as backtracking resolves things for you). Maybe you could try to slowly lower the backtrack number to see what fits for you? Another heavy consumer of time are the sub slot rebuilds; while you could disable that, it brings you back to having to run `revdep-rebuild` (which, as you might remember, takes up even more time). On a side note, work on Portage has revived; and there are some commits coming up that shave off a bit of time, so, I expect to see the time it costs drop towards the future... |
|
Back to top |
|
|
Josef.95 Advocate
Joined: 03 Sep 2007 Posts: 4601 Location: Germany
|
Posted: Sun Feb 23, 2014 2:15 am Post subject: |
|
|
You should dissolve the Qt libs conflict. (The output of the error message, however, is too short to be able to help)
.........................................................................
On my seven year old computer portage works fine.
(AMD Opteron Processor 185 2x2,6 GHz) Code: | portageq match / '*/*' | wc -l
1118
time emerge -pv world
....
real 0m56.484s
user 0m55.487s
sys 0m0.913s |
(sorry for my bad english) |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Sun Feb 23, 2014 4:36 am Post subject: |
|
|
patience is a virtue |
|
Back to top |
|
|
vladimir_ttt n00b
Joined: 22 Feb 2014 Posts: 2
|
Posted: Tue Feb 25, 2014 12:43 pm Post subject: |
|
|
I've solved the conflicts by with removing some packages to process.
But the main problem is performance.
I try it at work on system updated up to and and with 4 times faster CPU (i5-4670) and 16Gb
Code: |
real 2m38.606s
user 0m29.731s
sys 0m0.726s |
It is almost 3 minutes for answer that everything up to date!!!
I remember that the degradation has happened at one around 2-3 years ago.
It had worked perfectly before that.
when it has happened I asked my brother what could be reason for this?
He answer may be they are checking whole emerge tree checksums before giving the answer
( last part was a joke ) |
|
Back to top |
|
|
zsitvaij n00b
Joined: 11 Jan 2006 Posts: 74
|
Posted: Sat Mar 01, 2014 1:07 pm Post subject: |
|
|
The main performance killer is subslot tracking. It's an anti-feature that's on by default and offers little over preserved-libs and revdep-rebuild. The only things that come to mind are rebuilding maxima after an sbcl update and qt-core after an icu update, and those give very obvious signs of being broken when they are.
For comparison, on an i3-550@3.2 GHz:
Code: | # bsdtar -cf /dev/null /usr/portage /var/db/pkg |
(this makes sure all needed information is cached and not read from disk, so it doesn't factor into performance. bsdtar, because gnu tar knows that tarring to /dev/null is pointless and just reads the file descriptors and not the contents.)
Code: | # time emerge -pvuDN1 @world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB
emerge -pvuDN1 @world 727.34s user 6.94s system 99% cpu 12:16.08 total |
(now let's add '--ignore-built-slot-operator-deps y' to EMERGE_DEFAULT_OPTS in make.conf)
Code: | # time emerge -pvuDN1 @world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB
emerge -pvuDN1 @world 88.10s user 1.37s system 99% cpu 1:29.82 total |
This is with 709 packages in world. |
|
Back to top |
|
|
daddio Tux's lil' helper
Joined: 11 Oct 2003 Posts: 145 Location: Orem, UT
|
Posted: Fri May 02, 2014 4:26 am Post subject: horribly slow |
|
|
Performance truly has become horrific.
It seemed to become much worse about 2 years ago, and has steadily worsened since.
I hope we can get this fixed. Its deeply embarrassing to find dependency generation taking longer in many cases that the compiling.
zsitvaij, thanks for the heads up on subslot tracking... I don't know whether to look at the code, or just turn it off. |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Fri May 02, 2014 1:21 pm Post subject: |
|
|
If it automatically resolves more blocks than it used to (and it does), I'm willing to put up with the slowdown within reason. Execution time has gone up & down over the years as features have been added and then subsequently optimized. Nobody makes me sit around and watch the spinner until it's done. All that said, the chances of me switching to Ubuntu (or anything else) because Portage is slow (but reliable) are less than the chance that the proverbial tissue paper dog has of catching the asbestos cat on a romp through Hell.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Anon-E-moose Watchman
Joined: 23 May 2008 Posts: 6103 Location: Dallas area
|
Posted: Fri May 02, 2014 1:45 pm Post subject: |
|
|
time emerge -pvuD --with-bdeps y @world
These are the packages that would be merged, in order:
Calculating dependencies... done!
Total: 0 packages, Size of downloads: 0 kB
real 0m32.966s
user 0m32.296s
sys 0m0.340s
----
time emerge -pve --with-bdeps y @world
Total: 929 packages (929 reinstalls), Size of downloads: 38,588 kB
real 0m31.972s
user 0m31.247s
sys 0m0.378s
I've not noticed any large slowdown, perhaps something else is going on on your system.
Note: It will be a little slower after a fresh boot as nothing will be in memory cache. _________________ PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat May 03, 2014 6:36 pm Post subject: |
|
|
Consider trying a different package manager if Portage is failing you. There's at least two other options.
This is Paludis with 1019 packages and 6 overlays, doing the slowest thing you're likely to see in regular use, equivalent to $(emerge -DpvuN --with-bdeps=y):
Code: | ~ $ time cave resolve -c world
Done: 1 restarts, 5105 steps
These are the actions I will take, in order:
(nothing to do)
* No unread news items found
real 1m56.912s
user 1m10.908s
sys 0m1.474s |
This is slow, but it's always been this way and it's not getting any slower. |
|
Back to top |
|
|
yoshi314 l33t
Joined: 30 Dec 2004 Posts: 850 Location: PL
|
Posted: Thu Dec 04, 2014 10:21 am Post subject: |
|
|
There is also another option, reducing portage size.
Exherbo has small, topical, repositories. One for core system, one for x11, one for hardware support, etc ( list of major ones is here http://git.exherbo.org/summer/ ) . Typical user will install most of them, but it still beats having the whole shebang in one place.
Plus the VDB format seems more efficient, and calculating updates on the installation caches really well and this helps immensely, even if you change use flags in the meantime you still get improved resolution times due to caching of some common vdb files.
Note: this installation uses less repositories, so naturally resolves are faster. Also there is not much installed yet, system is on classic 7200 rpm hdd, on btrfs.
Code: |
# completely dropped i/o caches prior to this, as slow as it gets probably
time cave resolve installed-packages
real 0m31.955s
user 0m4.880s
sys 0m0.804s
# second run, cached
time cave resolve installed-packages
real 0m4.217s
user 0m3.940s
sys 0m0.208s
# re-run with more complete dependency check, cached
time cave resolve installed-packages -c
real 0m5.793s
user 0m5.496s
sys 0m0.192s
# rerun with validating installed package specs against available ones (if package definition changed, it's added to install queue)
time cave resolve installed-packages -km -Km -c
real 0m5.746s
user 0m5.484s
sys 0m0.184s
|
Even if you run different resolves or extensively alter use flags (called options on exherbo), it caches so well it affects future resolves greatly. The difference remains even after you enable lots of repositories and have a sizeable installed set.
On gentoo, resolves are significantly slower, but it can be attributed to size of portage package database. Caching effect is less visible even on machines with plenty of ram, and each resolve command pretty much takes the same amount of time, regardless of how many times you run it. _________________ ~amd64
shrink your /usr/portage with squashfs+aufs |
|
Back to top |
|
|
cwr Veteran
Joined: 17 Dec 2005 Posts: 1969
|
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Thu Dec 04, 2014 11:13 pm Post subject: |
|
|
yoshi314 wrote: | On gentoo, resolves are significantly slower, but it can be attributed to size of portage package database. Caching effect is less visible even on machines with plenty of ram, and each resolve command pretty much takes the same amount of time, regardless of how many times you run it. |
I was always under the impression that Paludis was slow because it's slow - no amount of struggling with cache options has made a significant improvement in the time I've used it.
That 10x speed difference between distros is pretty alarming. Makes me wonder what's actually missing in the Gentoo-side code... |
|
Back to top |
|
|
yoshi314 l33t
Joined: 30 Dec 2004 Posts: 850 Location: PL
|
|
Back to top |
|
|
v_2e n00b
Joined: 28 Dec 2008 Posts: 21
|
Posted: Fri Sep 22, 2017 7:51 pm Post subject: |
|
|
TomWij wrote: | Too much packages, too much overlays, too much unnecessary USE flags, ...; |
Could you please explain how the unnecessary USE flags influence the portage performance? Or maybe provide a link to the appropriate page? |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54420 Location: 56N 3W
|
Posted: Fri Sep 22, 2017 8:02 pm Post subject: |
|
|
v_2e,
USE flags control optional dependencies.
Sometimes, these are whole package trees in their own right, sometimes they are only options to the configure script.
When an unnecessary USE flag turns on a package tree, the dependency resolver has to resolve all the packages that are required by the USE flag.
As well as taking longer to resolve, these unnecessary packages will be compiled and installed on your system. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
|