Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Calculating dependencies in portage takes a (very) long time
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 3:11 am    Post subject: Calculating dependencies in portage takes a (very) long time Reply with quote

Hello,

I am having difficulty with portage right now. I have been experiencing issues (even when I upgrade portage) when trying to install packages. It takes 30 minutes to 50 minutes (Best educated guess), to calculate dependencies. I have tried deleting the edb/dep cache and emerging --regen and --metadata and no success. I have tried removing version numbers and other package names from /etc/portage/package.accept_keywords and checked for typos without any noticeable improvements. I have also tried python-cdb with many error messages such as parsing errors of the cdb.py file that I was supposed to create. Long story short, I had to uninstall python-cdb and restart konsole. I have looked on forums seeing if I could find a solution to my problem. I have not found a solution to this. By the way, on the portage upgrade, I had to add "sys-apps/portage" to package.accept_keywords to get emerge to detect the latest version of portage. The latest version emerge --search showed was 2.8-r1 or to that effect. After the package.accept_keywords atom was added to display version 2.11-r1 or so, I have installed it by using the nodeps flag to install portage and other packages. I am trying to be as thorough as possible when typing in all this information. I have upgraded portage by typing in:

Code:
emerge --nodeps portage


to get the latest version.

All help will be greatly appreciated.

Sincerely,

donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
alienjon
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1693

PostPosted: Wed Aug 06, 2014 3:32 am    Post subject: Reply with quote

I'm not sure that I've ever had to unmask an upgraded portage version. Is this an elective or required portage upgrade? If required, I'd suggest at least looking at the other masked/keyworded items and make sure nothing's wrong there (it just seems suspicious to me otherwise). Also, do you use layman? That'll slow you down a bit - especially if it's not setup properly (I'd suggest double checking the layman setup, if that's the case).

Also, and more fundamentally, how is the system running otherwise? If you're noticing general slowdown it could be hard drive read time (I've had drives act weirdly - odd slowing down, for example - if a drive's about to die).
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 3:38 am    Post subject: Reply Reply with quote

Hello,

I do not use layman. What I was trying to say was that because of the long calculating dependency times, I was unable to emerge portage by emerge portage. I had to use the --nodeps flag to get it to install.

Sincerely,

donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Wed Aug 06, 2014 3:49 am    Post subject: Re: Calculating dependencies in portage takes a (very) long Reply with quote

donald3.heckel wrote:
By the way, on the portage upgrade, I had to add "sys-apps/portage" to package.accept_keywords to get emerge to detect the latest version of portage. The latest version emerge --search showed was 2.8-r1 or to that effect. After the package.accept_keywords atom was added to display version 2.11-r1 or so, I have installed it by using the nodeps flag to install portage and other packages.

donald3.heckel ... you have keyworded the package and so are running the ~arch (or "unstable") version. This may introduce other issues as ~arch is where packages/changes enter the tree, so it wasn't so much a case that you weren't getting the current version (you were as 2.2.8-r1 is the current stable) but being prevented from installing a package that may (or may not) have issues.

Anyhow, does the following improve calculation time?

Code:
# emerge --ignore-built-slot-operator-deps=y <atom>

best ... khay
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 4:09 am    Post subject: Reply Reply with quote

Hello,

The option did not improve calculation time. Thank you for the suggestion.


Sincerely,


donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2605

PostPosted: Wed Aug 06, 2014 4:19 am    Post subject: Reply with quote

I know this isn't what you want to hear, but unless this happened suddenly you should learn to live with it. I'm guessing you're using older hardware. Is this correct?

Portage does several things when 'calculating dependencies' that takes a while to sort out. It can be frustrating, but it is important. There alternative package managers (paludis and pkgcore) which you may want to try. I have absolutely no personal experience with them and I can't say if it would help at all.

Using --nodeps is practically guaranteed to brake your system. You should basically never use it.

Also, if you don't want to guess how long it takes to run a command, ask the computer to keep track. For example, time emerge -av root
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 5:12 am    Post subject: Reply Reply with quote

Hello,

I see. Well, I guess it is time for me to switch package managers. Thank you very much for your input. What package manager do you recommend? I see the ones you have mentioned. Thank you for your input.
_________________
Believing is seeing.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7368

PostPosted: Wed Aug 06, 2014 5:22 am    Post subject: Reply with quote

Code:
time emerge -pv world
Total: 214 packages (20 upgrades, 1 downgrade, 4 new, 1 in new slot, 188 reinstalls), Size of downloads: 195,297 kB
real   0m53.047s
user   0m29.426s
sys   0m0.507s

time FEATURES="-colission-protect -preserve-libs" emerge -pv world --ignore-built-slot-operator-deps=y
real   0m25.598s
user   0m24.236s
sys   0m0.248s

Of course you will lost features, but if you don't care about preserved libs and collision, you can get good time saved.
And both could be replace with a revdep-rebuild run.

All those are options that are run on each package and/or each deps, i just prefer all of them disable and run revdep-rebuild when finish.

The calc is simple : suppose you have 10 packages and 3 will get broken.
portage must check the 10 packages to see if any libs will get broken. Once done emerge preserved-libs run will fix the mess (so the 3).
While i prefer portage ignore the 10 checks and just emerge them to get the 3 broken and i do one revdep-rebuild run (to fix the 3).

Not really the same result if your toolchain was one of the package in the broken list (with preserved-libs feature your toolchain is safe, while with revdep-rebuild you will not able to run it if your toolchain is broken).
But i must care about my toolchain because of distcc, so my toolchain is strictly control and backup (buildpkg feature is your friend)
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 1993
Location: Here and Away Again

PostPosted: Wed Aug 06, 2014 5:26 am    Post subject: Reply with quote

Teegrins, donald3.heckel!


You did not mention (I think) if this started to happen recently, or if it was fine before? What sort of hardware are you using?

Did you really mean to say that you spend 30 minutes to 50 minutes (sic) on merely calculating dependencies? I would imagine the hardware would need to be nearly ancient, or something very 'tiny' indeed. I have a laptop from 2005'ish, and it has no issues doing it in a jiffy (or at least a sort of a jiffy)...
_________________
Kind regards,
Chiitoo.

You might remember me from Gentoo projects such as Forums, LXQt, Qt, and Wine.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2605

PostPosted: Wed Aug 06, 2014 5:36 am    Post subject: Re: Reply Reply with quote

donald3.heckel wrote:
Hello,

I see. Well, I guess it is time for me to switch package managers. Thank you very much for your input. What package manager do you recommend? I see the ones you have mentioned. Thank you for your input.


I haven't tried any package managers other than portage, so I'm afraid I can't say. I think pkgcore might be the fastest but I don't have any data or experience to confirm this. But before you try switching, the options krinn pointed out are well worth trying.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 5:47 am    Post subject: Update Reply with quote

Hello,

How do I install pkgcore with python when it tells me "no module snakeoil found"?

Thanks

Sincerely,

donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7368

PostPosted: Wed Aug 06, 2014 5:48 am    Post subject: Reply with quote

i tried pkgcore, but in too early state to say it works
and i tried paludis, that might work, but everything is different from configs to flags. I don't have time to learn howto use that.
Code:
cave --help
usage: cave [ --environment class:suffix ] [ --log-level level ] COMMAND [ARGS...]
       cave help [ --all ]

Global Options:
  --environment, -E      Environment specification (class:suffix, both parts optional)
  --log-level, -L        Specify the log level
      debug (d)            Show debug output (noisy)
      qa (q)               Show QA messages and warnings only (default)
      warning (w)          Show warnings only
      silent (s)           Suppress all log messages (UNSAFE)
  --colour, -c           Specify whether to use colour
      auto (a)             Yes if stdout is connected to a terminal (default)
      yes (y)              Yes
      no (n)               No
  --help, -h             display help message
  --version, -v          display version information

See 'cave help [ --all ]' or 'man cave' for command names and options for those
commands
cave help --all
cave@1407303931: [WARNING portage_environment.dodgy] Use of Portage configuration files will lead to sub-optimal performance and loss of functionality. Full support for Portage configuration formats is not guaranteed; issues should be reported via trac.

Error:
  * In program cave help --all:
  * When making environment from specification '':
  * When creating PortageEnvironment using config root '':
  * PORTDIR empty or unset (paludis::portage_environment::PortageEnvironmentConfigurationError)

Beurk, looks like unfriendly noise to me.
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 5:55 am    Post subject: Reply Reply with quote

Hello,

My hardware is a dell inspiron mini 1018. My hardware is pretty decent with 1 gb of ram and an intel atom processor for a small laptop with 200 gb of hdd space. I currently with some recent upgrading and remodeling, have Linux Mint 17 Quiana (Used to be 14 Nadia), and Zorin OS 9 (used to be Ubuntu 12.10) installed alongside Gentoo. I primarily use my gentoo. My point of that is, I don't think that my hardware is that old. I believe the problem lies with portage. If any of you out there find any package managers, that I should try, I am open to suggestions.

Sincerely,


donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7368

PostPosted: Wed Aug 06, 2014 6:00 am    Post subject: Reply with quote

you have only 4 choices: portage, paludis, pkgcore or porthole (that actually use portage, but resolve itself dependencies, and do it a lot faster than portage).
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 6:05 am    Post subject: Reply Reply with quote

Hello,

Thank you very much. I think a frontend to portage would be very helpful. I will let you know of my progress.



Sincerely,


donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 6:15 am    Post subject: Reply Reply with quote

Hello,

I have successfully installed porthole. However, I get this error message when executing the program:

Code:
File "/usr/bin/porthole", line 27
    print "Porthole: python version = ", sys.version
                                      ^
SyntaxError: invalid syntax


Any suggestions?


Thanks

Sincerely,


donald3.heckel
_________________
Believing is seeing.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7368

PostPosted: Wed Aug 06, 2014 7:47 am    Post subject: Reply with quote

[ebuild R ] app-portage/porthole-0.6.1-r3 USE="nls" LINGUAS="fr -de -it -pl -ru -tr -vi" 937 kB
This version works here.
Back to top
View user's profile Send private message
donald3.heckel
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2013
Posts: 84

PostPosted: Wed Aug 06, 2014 4:32 pm    Post subject: Reply Reply with quote

Hello,

I am unsure how to install porthole, but it seems that this command flag might be a good workaround for portage:


Code:
emerge --backtrace=0 <packagename>



I am still unsure on how to install porthole. On sourceforge, The ebuild downloaded as a .bin file and it gave me a syntax error when executing the program.

Sincerely,

donald3.heckel

Update: I believe portage did not detect a dependency that was already installed for porthole. I have already installed atk from source using:

Code:
./configure --prefix=/usr && make && make install


yet portage did not discover it. So I then did:

Code:
emerge --nodeps porthole


because I knew that the dependency was installed from source to my /usr directory.

I believe porthole is installed. I will let you know of my progress.
_________________
Believing is seeing.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Wed Aug 06, 2014 5:57 pm    Post subject: Re: Reply Reply with quote

donald3.heckel wrote:
Update: I believe portage did not detect a dependency that was already installed for porthole. I have already installed atk from source using: ./configure --prefix=/usr && make && make install

donald3.heckel ... and why should it? How can you expect portage (or any package manager for that matter) to "manage" software if you install software outside of it? /usr is for package installed software only and installing there will inevitably lead to issues. Also, having done so it makes it impossible to support because we simply can not say what the current state of your system (dependencies, etc) is ... I could hypotheses that the reason it takes "takes 30 minutes to 50 minutes [...] to calculate dependencies" is because your install it totally broken due to similar arbitrary software management.

What's more 'atk' is available as a package ... so there was no reason to install "from source".

Code:
# eix -eI# atk
dev-libs/atk

donald3.heckel wrote:
Code:
emerge --nodeps porthole

The Doctor already warned you about the danger, and inevitable outcome, of using '--nodeps'.

best ... khay
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7368

PostPosted: Wed Aug 06, 2014 6:50 pm    Post subject: Reply with quote

I highly recommend to forget what i said upper and not disabling preserved-libs, collision protect & friend with how you use it.
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Wed Aug 06, 2014 7:03 pm    Post subject: Re: Reply Reply with quote

donald3.heckel wrote:
Code:
emerge --backtrace=0 <packagename>

This should be
Code:
emerge --backtrack=0 <packagename>

Actually I'd recommend to add this option to EMERGE_DEFAULT_OPTIOINS.


About available package managers, there are only two true alternatives to portage. Pkgcore was abandoned by it's original maintainer and only recently Radhermit took up the ball. Currently pkgcore isn't working with the tree but should be back in business soon. Pkgcore is supposed to be a lot faster than portage. I used pkgcore primarily for its pquery command in the pre EAPI 5 days. Paludis is the other option. Portage config format is supported but only poorly. The native config format you need to get used to first but is vastly superior to Portages, well worth to have a look at at least once. About the cryptic output, let's say the last 5 years Portage output started to look a lot more like Paludis. It might be overwhelming at first but is crucial to understand what the package manager wants to do. People usually mean with fast the time it takes to calculate an update. There Paludis isn't fast either but it's doing quite a bit more validation work than Portage does. There are other things you want a package manager to do. Paludis is sometimes orders of magnitude faster. In terms of features you lose some, you gain some.

Generally the best results you get by switching your HDD with an SSD :wink:
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3345
Location: Gainesville, Florida

PostPosted: Thu Aug 07, 2014 5:31 pm    Post subject: Reply with quote

Sorry to say, but IMO your Dell Inspiron Mini 1018 with only 1GB of 667mhz ram and the 1.66 Hz Intel Atom one-core cpu is seriously underpowered to decently run a source distro like Gentoo. Even if you have a large swap partition, it's probably not going to help that much. If you are loading any other apps except the OS, you are likely using up all your ram almost immediately after booting.

The most likely reason your portage is so slow is that it has to constantly operate everything via the swap partition, and worse yet, if you have a too small swap space the OS will have to calculate which process has priority, and which process needs to be (or is safe to) "ejected" from swap. All this extra processing adds considerable time and complexity, and is too much to ask of a one-core cpu functioning with only 1GB of 667mhz ram. Thus, things crawl to a snails pace, if they complete at all.

I would suggest that when you need to run portage, boot to a "bare minimum" state (no X, just use terminals), and run your updates. It might make a little difference, but I'm not optimistic. I'd explore the possibility of adding ram, or replacing the cpu with a dual core to the nspiron Mini 1018.

OK- got curious and just checked online- you can only upgrade ram to 2GB, no cpu upgrade possible. Links below. However, in my opinion without a cpu upgrade it's not worth it. Probably only one ram slot, so you'd need a 2GB stick. I looked at the video link- seemed like a PITA to replace the ram, and I've built many systems myself.

Bottom line- if you really want to run Gentoo, with all the regular compiling required, I'd consider newer hardware. Even a low-end cheap used system with at least a 2-core cpu and 2-4GB ram would make a world of difference.

http://en.community.dell.com/support-forums/laptop/f/3518/t/19564460.aspx

https://www.youtube.com/watch?v=Iy-VaSKEQqg
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r6, gcc-9.3.0 kernel-5.5.11-gentoo USE=experimental
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10456
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Aug 07, 2014 5:49 pm    Post subject: Reply with quote

Timings on a 2.8GHz i7 with way too much RAM:
Code:
~ # time emerge -pv @world
...
Total: 265 packages (1 upgrade, 1 new, 263 reinstalls), Size of downloads: 1,231 KiB

real    1m8.813s
user    0m40.438s
sys     0m0.785s
I have an Atom at home with (I believe) similar specs that isn't nearly as slow as the OP is reporting. I'll post a similar run on that for comparison this evening.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3345
Location: Gainesville, Florida

PostPosted: Thu Aug 07, 2014 6:11 pm    Post subject: Reply with quote

John,
Good to know. Maybe I'm mistaken, and it's not entirely a hardware issue as I suspected.
Please include the specs of your atom along with the timed run. I'm really curious as to figure out what is actually causing the drastic slowdown on the OP's system.

However, I'm still of the opinion that the OP is still in dire need of a serious hardware upgrade. :wink:

EDIT: John,
The Dell Inspiron Mini 1018 came out in late 2010, with a 1.66GHz cpu, and intel lists these for 2010, so OP's is likely one of these.

http://ark.intel.com/products/42503/Intel-Atom-Processor-N450-512K-Cache-1_66-GHz (1 core)

http://ark.intel.com/products/43098/Intel-Atom-Processor-D510-1M-Cache-1_66-GHz ( 2 cores)

http://ark.intel.com/products/43517/Intel-Atom-Processor-D410-512K-Cache-1_66-GHz (1 core)

http://ark.intel.com/products/49491/Intel-Atom-Processor-N455-512K-Cache-1_66-GHz (1 core)

Theres about 50 other atom cpus listed on the intel page: http://ark.intel.com/products/family/29035/Intel-Atom-Processor#@All

Many are more recent 1 and 2 cores, and even one quad core. Perhaps yours is a 2 core, explaining the better performance you report?
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r6, gcc-9.3.0 kernel-5.5.11-gentoo USE=experimental
Back to top
View user's profile Send private message
Cyker
Veteran
Veteran


Joined: 15 Jun 2006
Posts: 1746

PostPosted: Thu Aug 07, 2014 7:41 pm    Post subject: Reply with quote

I used to run Gentoo on a K6-III 400 and it never took anywhere *near* 30 mins just to calculate the dependencies.

There is something wrong with the OP's system.

If it happened suddenly, I would check the hard disk first.

If it's been getting slower over time, there maybe a file or database that's corrupt.

I must admit, I don't know where to start looking tho'!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum