Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Gentoo linux okbu not gentoo linux actually yes gentoo linux
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  

Is there a more effective way to install than emerge?
Yes
10%
 10%  [ 1 ]
No
90%
 90%  [ 9 ]
Total Votes : 10

Author Message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 886

PostPosted: Tue Jul 10, 2018 5:15 am    Post subject: Gentoo linux okbu not gentoo linux actually yes gentoo linux Reply with quote

I would like to know how to save all that dependency calculation work that Portage does in order to minimize time when running --pretend or --ask (followed by a no) and not actually merging anything. Why does Portage not act smart later when I want it to do the same function and using the same system environment, tree for sources, and everything else like that? Why does it go through all those calculations again in order to merge stuff that could have been saved to some sort of cache?

Does anyone have any hints for creating such a cache that would save the "state" of Portage in the midst of building (say several hundred packages), or else just prior to that?

Motivation: If sometimes I am getting ready to leave the house and I run the command, so I had to wait to see what the output would look like. Now later I come home and want to review the list of packages maybe read up on them, and then ultimately merge them. That is up to three times that I might have to run that command.

If Portage takes 3 minutes each time in figuring it out, those add up and pretty soon it becomes a drag or at least discouragement for trying different cases or scenarios (like use flags, environment variables, etc.).

Not to mention the times when I maybe meant to press "y" and instead "n" after emerge --ask and now i have to wait for portage to figure all that stuff out again?

My suggestion would be either to help users with a smarter interface (no i don't mean flashy or shiny buttons although i don't see how that would hurt), but what I mean is if you aren't a pro system admin, which I am not, if you aren't a developer (I am really not) then it seems like the only way to fix problems in terms of troubleshooting Portage is to turn to people on this forum for help. I really appreciate the help I get here, but at the end of it all when a package is failing to build or portage won't move forward there seems like a pretty large gap there for what the package manager could do in terms of presenting the user with some viable alternatives (either helpful or non-helpful, but by presenting several options that could be a better case scenario than just some generic statement about Build Conflicts and how those can get resolved, etc.)

This is my rant. I don't mean to take it out on any of the hard working developers who built the package management system...that is hardly my goal, but I want to continue getting the benefit just without some of the "slow downs" that are provided by emerge. I remember seeing some pretty cool scripting ways to work with the system update, like with eix and then piping cmmands to download and install packages from source. If anyone has some helpful scripts for that a URL explainaing a different way to handle updates on my Gentoo system that doesn't have me sitting around more often than not waiting for Portage output...then I would like that!

Maybe it would also hellp me to get clear on some of the emerge options like the --pretend option just in case I can't be sure what it does other than to run conflict checks and stuff like that. But I don't think it will be able to account or predict for the situation of a package that won't build because of some other missing library or something like that. Is that what other package managers are for? I haven't tried any others within Gentoo other than Portage. I know that there should be a way to just create my own ebuilds or not even have to and just use ebuild command...right for the more advanced interaction am I on target with that last statement?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41404
Location: 56N 3W

PostPosted: Tue Jul 10, 2018 11:54 am    Post subject: Reply with quote

Moved from Off the Wall to Other Things Gentoo.
Its not really OTW material and it is a reasonable technical question too.

I voted No ... the alternative is dependency hell, where you manage all the depenencies yourself.
Binary distros can do it as they only give you one choice - take it or leave it.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10118
Location: Somewhere over Atlanta, Georgia

PostPosted: Tue Jul 10, 2018 12:24 pm    Post subject: Reply with quote

"No" also for me. It's been discussed before, but there are issues with caching the dependency graph. Just to mention a couple:
  • If you change a USE flag, then a significant amount of the dependency graph might need to be recalculated.
  • If you emerge a new package, then the cached dependency graph will be incomplete as the new package will most likely bring new dependencies into the graph.
- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12802

PostPosted: Wed Jul 11, 2018 2:45 am    Post subject: Reply with quote

I would love for the system to be better at caching results, but it is a hard problem. If the cache manager is not aggressive enough at invalidating stale results, you get weird and confusing output when the system reuses data that should have been discarded (for example, what if Portage's cache sometimes incorrectly ignored new packages being installed, so you got an output saying you need to install Foo, but you just finished installing Foo?). If the cache manager is too aggressive at invalidating stale results, you get little benefit from the cache. Most people won't be rerunning exactly the same command, with no intervening USE flag changes, mask changes, other installs, etc., so the obviously safe cache invalidation solution (discard on any system state change) will usually not help. To get good results, you quickly descend into very selective invalidation rules in an attempt to invalidate as little as you can. That is not only difficult to get right, but fragile since it depends on assumptions about what types of system state change are relevant. For example, consider the mess if the cache were developed before the invention of package USE dependencies, and therefore the invalidation algorithm assumed (correctly at the time of its invention, but incorrectly now) that changing the USE flag on Foo cannot impact the dependency resolution of Bar.
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 886

PostPosted: Sat Jul 14, 2018 8:34 am    Post subject: Reply with quote

I seem to get the idea that caching the output is going to be very complex and perhaps unnecessarily so. But maybe a different approach rather not caching it but somehow saving the amount of time that Portage calculates some commands that are similar relatively given a nearly exactly similar system. Now I get that the tree is going to always be the reason for recalculation but there just doesn't seem to enough changes from one day to the next in the tree to not be able to consider some realistic time saving in calculations perhaps if even based on a cached tree??

Anyway, I've been using Gentoo for a while now and I did want to test out some other package management options so now may be the time.

Would you recommend going the route of actually installing another PM or else just make some attempts to use the built in Portage tools like ebuild to start managing the packages a little differently. If there is a clear shortcoming for me it is the concept of sets because I can't seem to get good working use out of any of them other than 1 or 2 of them (system and world)...and I don't really get what the point of those are most of the time since --deep like some other flags seem to "hide" a lot of the work that is done and make the resulting output confusing.

Basically I mostly just run into the updates with world or system (to save time)...but now I am thinking what would save time might be to write some scripts of my own that could query if those updates are even available and then go from there.

I assume that the package management options for Gentoo are somewhat limited (Paludis, Portage, are there more?) but with Portage at least I can hopefully begin to get used to installing packages from source just like emerge would but with ebuild instead.
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6204
Location: Room 101

PostPosted: Sat Jul 14, 2018 9:17 am    Post subject: Reply with quote

LIsLinuxIsSogood wrote:
If there is a clear shortcoming for me it is the concept of sets because I can't seem to get good working use out of any of them other than 1 or 2 of them (system and world)...

LIsLinuxIsSogood ... if you look at the output of 'emerge --list-sets' then you'll see sets other than @system and @world. So, something like @x11-module-rebuild can be used when x11 is updated, and rebuilding 'modules' (ie, xf86-input-*, xf86-video-*) is required. There is also @preserved-rebuild which can be used for rebuilding packages against an updated library but who's the previous version was retained because some package in @preserved-rebuild was built against that version.

LIsLinuxIsSogood wrote:
Basically I mostly just run into the updates with world or system (to save time)...but now I am thinking what would save time might be to write some scripts of my own that could query if those updates are even available and then go from there.

I very much doubt this will improve the situation, the PM will still need to calculate the dependency graph.

BTW, one way to reduce calculation time is to add '--ignore-built-slot-operator-deps=y' to the command, or EMERGE_DEFAULT_OPTS.

LIsLinuxIsSogood wrote:
I assume that the package management options for Gentoo are somewhat limited (Paludis, Portage, are there more?) but with Portage at least I can hopefully begin to get used to installing packages from source just like emerge would but with ebuild instead.

plaudis was given last rites, and so the only other option is sys-apps/pkgcore ... this is faster but lacks some features available in portage.

best ... khay
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41404
Location: 56N 3W

PostPosted: Sat Jul 14, 2018 10:13 am    Post subject: Reply with quote

LIsLinuxIsSogood,

Using ebuild to install packages is like
Code:
emerge --nodeps <package>

That makes you the package manager.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6839

PostPosted: Sat Jul 14, 2018 10:16 am    Post subject: Reply with quote

you can also cut down calculations if you tell portage there's just nothing to calculate, just remember to properly use onehost

emerge -p packagA (getting the list)
packageB
packageA...
emerge packageA packageB.... --nodeps --oneshot (no more busy)
ps: like what NeddySeagon said, but slower
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 886

PostPosted: Sat Jul 14, 2018 10:58 am    Post subject: Reply with quote

I'm going to install pkgcore on my desktop pc, which is where I run stable packages so less likely to have issues there I suppose than on my virtual Gentoo Box that houses pretty much double the packages of that one.

In response to sets the @preserved-rebuild set is something that I still struggle with in terms of deciding what it is that should be done.

E.g. right now I have a package listed in there with a dependency that has since been updated but that I had patched in order to prevent some gui problems that abiword had on that version. I've been reluctant to upgrade since without a separate slot in existence for the package in gentoo I would either have to build from source or create a "shell" of ebuild with new slot to be able to do that. Either way it is a lot more work. So in a way in regards to comparing different PM and pkgcore missing slot dependency check seems like in this case Portage itself has no real advantage. Unless there is some easy command that I am unaware to install a secondary or third build of the same package on the system and have Portage keep track of each one. But I assume that isn't the purpose of the package manager in Gentoo.

Neddy is right and I like the idea of being the package manager for my system, but I guess the point would be to build in to the situation a certain fail-safe on my own calculations. There seems to be plenty in Portage to do this, so I think I will stick with that for now.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41404
Location: 56N 3W

PostPosted: Sat Jul 14, 2018 11:09 am    Post subject: Reply with quote

LIsLinuxIsSogood,

You know that portage can apply patches for you?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 886

PostPosted: Sat Jul 14, 2018 11:25 am    Post subject: Reply with quote

Yes that is how I managed to fix the application in the first place. My concern is the patch could be different for the newer package and I haven't checked (just lazy) but I guess I would need to find out before applying it to the version bump...is that right?

The real problem with Portage as a package manager isn't the functionality really it is that it just is more than I need these days.

UPDATE: The file looks good without the patch, from what I can tell with the version bump of abiword to 3.0.2-r1 no screen flickering which was an issue that got in the way about 6 months ago or 1 year.

I initially enjoyed aspects of customizing to the highest degree possible with portage, but more and more my needs these days are actually becoming common given a better foundational understanding of more tools to get the job done, for example application space, userland(?) and kernel tools, all being able to work together even gui in order to fix a rough problem with a system.

But if there is something that really doesn't play in much at all to the experience of a working system it is package management. That is why I started this thread, and would like to know about what other tools or workarounds exist so that I can continue to keep my Gentoo boxes up and running, smoothly with as little effort possible. For really many years now I have been able to basically rely on occasional updates, and a very rare running of --depclean. That's prety much what I've been doing for years now and it seems to work, but would be nice to have some other options. So I think I will be trying pkgcore on at least one of the systems.

DISREGARD BELOW THIS LINE SINCE THE UPGRADE WORKED TO ABIWORD-3.0.2-r1
Any recommendations in terms of checking if a patch will work before upgrading, or else keeping the previously patched version so that in case things go wrong I can always revert back to the patched one. That seems fairly easy at least (I think)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41404
Location: 56N 3W

PostPosted: Sat Jul 14, 2018 12:41 pm    Post subject: Reply with quote

LIsLinuxIsSogood,

Read up on pkgcore before you deploy it. It is known to be feature incomplete.
Does it support EAPI 7?

You should use --depclean every update. If you don't, portage will maintain the orphaned packages on your system until you do depclean them.
That's extra time to calculate the dependency tree, build and install packages that you don't actually need.

You should also check your world file for pollution from time to time.
--depclean will not remove any packages that are listed here but its all too easy to forget --oneshot when you are looking into a build failure.
This pollution can keep things around that you don't need ... and the dependencies.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6839

PostPosted: Sat Jul 14, 2018 2:58 pm    Post subject: Reply with quote

NeddySeagoon wrote:
That's extra time to calculate the dependency tree, build and install packages that you don't actually need.

No NeddySeagoon, that's just 0.00% extra time ; dependencies/program that are useless takes nothing from portage because they are just not pull in the graph at all. But they do take space.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41404
Location: 56N 3W

PostPosted: Sat Jul 14, 2018 3:01 pm    Post subject: Reply with quote

krinn,

Well caught. I was thinking ahead to extra entries in @world.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
LIsLinuxIsSogood
l33t
l33t


Joined: 13 Feb 2016
Posts: 886

PostPosted: Sun Jul 15, 2018 2:41 am    Post subject: Reply with quote

How best to do this world set cleanup so to speak? Obviously --depclean should work, but I've had past scenarios before I really knew how to fix it where running that command often created a lot of weird circular dependencies.

Maybe there is another wya?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41404
Location: 56N 3W

PostPosted: Sun Jul 15, 2018 10:05 am    Post subject: Reply with quote

LIsLinuxIsSogood,

Cleaning world consists of removing entries that should not be there.
There is a wiki page and a discussion cleaning the world file (wiki) - check the script

Essentally, for every entry in the world file, check that nothing depends in it. If the package is pulled in as a dependency, it should not be in world.
That gets a list of candidates for removal. You cannot just remove things, its your system.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10118
Location: Somewhere over Atlanta, Georgia

PostPosted: Sun Jul 15, 2018 3:32 pm    Post subject: Reply with quote

LIsLinuxIsSogood wrote:
How best to do this world set cleanup so to speak? Obviously --depclean should work, but I've had past scenarios before I really knew how to fix it where running that command often created a lot of weird circular dependencies.
Just to clarify, --depclean doesn't clean the world set at all. What it does do is to remove installed packages that no other installed package depends on.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum