Gentoo Forums :: View topic - Why has portage become so slow

Why has portage become so slow

View unanswered posts
View posts from last 24 hours

Goto page Previous 1, 2, 3, 4, 5, 6, 7 Next

Gentoo Forums Forum Index

Portage & Programming

View previous topic :: View next topic

Author

Message

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Thu Oct 31, 2013 4:41 am Post subject:

ArneBab wrote:

I keep using and enjoying Gentoo, though my time is much more limited these days. Luckily I could choose to install my own system for doing my PhD (the institutes admin said “if you maintain it yourself”), so that I am now using Gentoo at work, too

Heh lucky you :-) What is your thesis about?

Quote:

I hope pkgcore will get active again! Sadly I could not find new stuff in its commits…

Yeah istr radhermit is working in his own repo: I'm sure he'd love someone with your experience testing it, provided you were in #pkgcore and not filing bugs against work-in-progress, which would just slow everything down. (For that reason, I'm not going to dig out the url from my logs, otherwise I would ofc.) I think things get pushed relatively quickly, but I'm too old to remember last week, when memories from 30 years ago are so much more vivid. ;-)

_______0
Guru
Guru

Joined: 15 Oct 2012
Posts: 521

Posted: Thu Oct 31, 2013 1:59 pm Post subject:

steveL wrote:

TomWij wrote:

steveL wrote:

Whereas the forums are supposed to be more informal.

That's your opinion, which your whole posts bases itself on as well as try to back it up; it clearly opposes to my opinion, which my previous posts bases themselves on as well have backed that up. Now, who decides what the forums are to be? Neither of us. So, please stop telling me what to do or addressing me for what doesn't match your opinion, we both are not convinced of each other's approach and I'm not sure if we will ever be...

Fine let's leave it there then; just bear in mind I'm not the one throwing my weight around, reporting people left, right and centre, and causing several other users to complain about trolling.

Again, I do support your position in general, often find myself agreeing with you on the mailing-list, and have always had a great deal of respect for you, based on your conduct on IRC and the help you give users.

Can we at least agree that crap code is crap code whoever writes it? </rhetorical;>

I am not an expert nor do I know how to code.

However I do recognize ppl's post for their technical merit and it's quite fun and interesting in Gentoo.

This dude trigger-happy with his systemd dildo is infecting every post he touches. Quite frankly, his un-imaginative dissective posts make mostly non-sense, don't add or address any details to the subject, don't help to resolve anything. I don't know who planted this troll with a dev badge but it's a mistake. All the years that I've visited the forums this dude is the weirdest shit happening in the forums.

I just want to confirm that I too view this dude as troll, and support all those that are voicing patiently and very politely their displeasure towards the troll. He is interfering with experience ppl and has no respect towards them. The troll is even making other ppl get banned.

plz, some admin temporarily ban the troll, it's gonna do him good.

Anyways, I too don't want to get drawn out with bicketing, but I find it sad what's happening.

ps.: Dear Troll, don't dissect this post, I ain't feeling yo systemd dildo and learn how to organize sensical thoughts. (unless you get paid for trolling)

ArneBab
Guru
Guru

Joined: 24 Jan 2006
Posts: 429
Location: Graben-Neudorf, Germany

Posted: Thu Oct 31, 2013 2:37 pm Post subject:

steveL wrote:

Heh lucky you :-)

What is your thesis about?

Inverse modelling of sources and sinks of greenhouse gases - using Python and Fortran, coordinated with Mercurial.

steveL wrote:

Yeah istr radhermit is working in his own repo

I can use Google, so the name suffices ☺

Thanks!
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.

ArneBab
Guru
Guru

Joined: 24 Jan 2006
Posts: 429
Location: Graben-Neudorf, Germany

Posted: Thu Oct 31, 2013 3:15 pm Post subject:

ArneBab wrote:

steveL wrote:

Heh lucky you :-)

What is your thesis about?

Inverse modelling of sources and sinks of greenhouse gases - using Python and Fortran, coordinated with Mercurial.

steveL wrote:

Yeah istr radhermit is working in his own repo

I can use Google, so the name suffices ☺

pkgcore is working again!

It’s so fast that I can quickly test stuff again. Thank you! And thank you, radhermit!
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Fri Nov 01, 2013 7:47 pm Post subject:

ArneBab wrote:

Inverse modelling of sources and sinks of greenhouse gases - using Python and Fortran, coordinated with Mercurial.

Interesting; yeah Fortran is still used a lot for heavy mathematical calculations. I really should explore interfacing with it from C, when I get some time.

Glad pkgcore is starting to get there: I'll try radhermit's version with update (and emerge=pmerge in /etc/update) when I can. It'll certainly stress-test the portage output-formatter. ;)

@__O: leave it, man; I was so glad to finally get onto a new page and not deal with this BS any more. You've expressed your opinion, as is your right. Don't get into it any more, even if someone else should reply, as trolls have a habit of using your frustration against you, typically to claim it's a violation of the Code of Conduct, to be annoyed by their trolling, and sometimes even just to respond to the points they make. And it just won't stop, so:
Don't feed the trolls.

Sometimes I really wish the forums had an /ignore like IRC. That way I could switch people out for a week or two (except when someone else replies to them) and not get into trouble for taking the time to respond. It's just hassle, and a real downer, ime.

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Fri Nov 01, 2013 8:06 pm Post subject:

TomWij wrote:

I wonder how much of the tree really uses the new EAPI though; it might be interesting to see how we can convert back EAPI 5 ebuilds to EAPI 4 ebuilds and just have it work again until the new EAPI is implemented, we can then do the same with EAPI 6 later on.

That's a crap idea, imo. It adds a lot of work for no real benefit, when we all know the EAPIs are already approved by the Council (or will be in the future.)

Now if we stopped adding stuff to EAPIs that required difficult-to-implement changes in the mangler, and specified we're only adding BASH functionality for the next one, that would give time for pkgcore to catch up (which it appears close to, already) and additionally give Gentoo time to switch to focussing on pkgcore, which resolves the entire subject of this forum topic.

TomWij
Retired Dev

Joined: 04 Jul 2012
Posts: 1553

Posted: Fri Nov 01, 2013 8:50 pm Post subject:

steveL wrote:

That's a crap idea, imo. It adds a lot of work for no real benefit, when we all know the EAPIs are already approved by the Council (or will be in the future.)

Just a rewriter could be done in a reasonable amount of time; but yeah, it does require extra work and doesn't really fix the "run behind" problem.

So, now what about if we let pkgcore "ignore" the new syntax; this could throw an "not yet implemented" error for developers, but be silently ignored for users. The parsing has to be written anyway; so, as a consequence you don't have to do any extra work with this approach. The only thing that not fits in the picture are packages that explicitly rely on an EAPI 5 feature in order to build; but well, that's something the developers could focus on implementing first such that this last thing will no longer be a problem.

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Fri Nov 01, 2013 11:19 pm Post subject:

TomWij wrote:

Just a rewriter could be done in a reasonable amount of time; but yeah, it does require extra work and doesn't really fix the "run behind" problem.

Put it this way: there is zero benefit, and it requires work. Thus the reward to work ratio is er zero.

Quote:

So, now what about if we let pkgcore "ignore" the new syntax; this could throw an "not yet implemented" error for developers, but be silently ignored for users. The parsing has to be written anyway; so, as a consequence you don't have to do any extra work with this approach.

Why bother? I'd just put the features in as you add the logic to parse them, one by one till all the ducks are in a row. It's more fun working like that as well, unless you have someone specifically working on the parser alone, in which case throwing the unimplemented exception is fine: it tells the other developers what they haven't done yet, and decouples the work. Otherwise no, bad idea imo, since keeping the sense of fun and achievement is vital to keeping people motivated. That's another reason why incremental development is useful. And in no circumstance would I push a release of pkgcore until EAPI5 were complete to my satisfaction, if I were radhermit, whom I have quite a deal of faith in atm; that can change ofc, it's changed with several other devs before now.

No, that does not mean I am saying incremental development is the only method (before you start shooting down my statements on those grounds): a balance in all things, is required. Incremental development does not mean kludge on top of kludge: it means finishing one thing properly before you go on to the next. And not being afraid to rework what you have literally just written and think is the bees-knees, or throw it away and start again: which requires one not to be too precious about one's work. Without critical self-evaluation, a "developer" is not a programmer. Nor can they ever be.

At best they will be a nightmare to work with, at worst they will use propaganda and marketing instead of technical excellence to push their work on others who know it's rubbish, based on long decades of hard-won experience, or simply on snafu after snafu from field-testing, always accompanied by some bs excuse, blaming someone else, as ever. Even if the problem they're blaming is their own work (insight is not a strong point of such people.)

The fact that experience comes from making mistakes, does not stop them being mistakes. If you don't learn from your mistakes, but instead blame everyone else for them, you don't gain any experience: you just become a worse "developer" until the only people you have around you are sycophants who drank the kool-aid, and reinforce the idiotic perception that "users are the problem."
IME developers make the classic user mistakes far more than users do, since they start off with the idea that they must know what they are doing, even if they have no experience nor competence in the problem-domain. WAYTTD and X-Y problems are far worse as a result, and it's much harder to get them to realise that it is an X-Y problem.

Users on the other hand tend to be delighted when you tell them "you don't need to do all that malarkey, <util> was written to do exactly that" since they are interested solely in getting the job done, as efficiently as possible. Devs tend to take it as a snub on their "brilliance" unless they've got past that. In which case they love it.
I certainly do: if someone shows me a way to throw away a load of code and do it ten to a hundred times faster, I pay them respect.

Again, just my opinion, and not directed at any individual, before you report me yet again. Even if it were, it would only be a description and critical evaluation of behaviour and work method, not a personal attack, btw. And if someone can't handle that, afaic they need to grow up, and see/consider how they get treated in the real world of work.

TomWij
Retired Dev

Joined: 04 Jul 2012
Posts: 1553

Posted: Fri Nov 01, 2013 11:45 pm Post subject:

steveL wrote:

Again, just my opinion, and not directed at any individual, before you report me yet again. Even if it were, it would only be a description and critical evaluation of behaviour and work method, not a personal attack, btw. And if someone can't handle that, afaic they need to grow up, and see/consider how they get treated in the real world of work.

Don't worry, understood it to be general; just want to leave a note that in the real world of work there are customers and deadlines, you can't just delay pushing a release under such conditions. Or well, at least not with without considering to evaluate the possibility through risk analysis first. Part of that applies to pkgcore as well; because if you don't release it in a reasonable amount of time and keep people waiting, they'll simple drop interest into learning and using a package manager that breaks with the next EAPI. In other words "Release early. Release often. And listen to your customers" ~ Eric S. Raymond (http://en.wikipedia.org/wiki/Release_early,_release_often) which does not conflict with incremental development; you just split up the tasks in smaller actionable and achievable parts, it'll still be incremental.

While "fun and achievement" is vital, it is also rather limited at the work place because of customers you have to listen to and deadlines you have to meet; and that is why at least half of that (the "fun" part) is for what you get to do when you get back home from work.

desultory
Bodhisattva

Joined: 04 Nov 2005
Posts: 9410

Posted: Sat Nov 02, 2013 4:49 am Post subject:

_______0 wrote:

This dude trigger-happy with his systemd dildo is infecting every post he touches. Quite frankly, his un-imaginative dissective posts make mostly non-sense, don't add or address any details to the subject, don't help to resolve anything. I don't know who planted this troll with a dev badge but it's a mistake. All the years that I've visited the forums this dude is the weirdest shit happening in the forums.

I just want to confirm that I too view this dude as troll, and support all those that are voicing patiently and very politely their displeasure towards the troll. He is interfering with experience ppl and has no respect towards them. The troll is even making other ppl get banned.

plz, some admin temporarily ban the troll, it's gonna do him good.

Anyways, I too don't want to get drawn out with bicketing, but I find it sad what's happening.

ps.: Dear Troll, don't dissect this post, I ain't feeling yo systemd dildo and learn how to organize sensical thoughts. (unless you get paid for trolling)

Regardless of your personal opinion of other individuals on the forums, you are expected to maintain a reasonable level of decorum when expressing those opinions here and invoking contaminated sex toys fails that test rather handily. Consider this a warning that you should take more care in expressing your opinions.

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Sat Nov 02, 2013 7:57 am Post subject:

TomWij wrote:

I think you'll find that very little risk analysis happens under such circumstances: typically your manager is a complete and utter toe-rag who has NFC what implementation is about, and people are shouting at each other, losing sleep, and generally burning out. Mismanagement is far more common than you might think. "Executive thinking" is an oxymoron ;) Typically management just shouts whenever things aren't going how they'd like, and because that sometimes gets people to work (rather than face the tirade, since they're not allowed to say what they think) managers kid themselves that it's effective. It's not in the longer-term, but short-termism is endemic in corporations: you're only as good as your last quarterly, and "we can just hire more programmers, right? Anyone can program, my nephew could do it for less."

There's exceptions to every rule of course, and there's more emphasis placed on "looking after the programmer" nowadays: but it's just hype when the chips are down; and you certainly are very expendable according to this world-view (and to every corporation, in fact: that's why Trade Unions were fought for, and died for, over centuries.) So's everyone else, which does not make for a happy atmosphere, and instead leads to people politicking and using procedures and policies to coerce the situation to one of their liking. Of course that doesn't work either, since everyone is playing the same game, and everyone knows it. They just know what words to parrot to sound like they mean it.

Hence buzzwords and bulshytt:

Code:

Bulshytt: A technical term denoting speech (typically but not necessarily commercial or political) that employs euphemism, convenient vagueness, numbing repetition, and other such rhetorical subterfuges to create the impression that something has been said.

Quote:

Part of that applies to pkgcore as well; because if you don't release it in a reasonable amount of time and keep people waiting, they'll simple drop interest into learning and using a package manager that breaks with the next EAPI.

We're way past that point, in case you hadn't noticed ;p.

Quote:

While "fun and achievement" is vital, it is also rather limited at the work place because of customers you have to listen to and deadlines you have to meet; and that is why at least half of that (the "fun" part) is for what you get to do when you get back home from work.

You're a student, aren't you?
Anyhow, that's exactly why we like an informal forum, as I already explained on IRC when I got in touch with you a week ago.

TomWij
Retired Dev

Joined: 04 Jul 2012
Posts: 1553

Posted: Sat Nov 02, 2013 10:11 am Post subject:

steveL wrote:

We're way past that point, in case you hadn't noticed ;p.

Yet that doesn't prevent it from happening again.

steveL wrote:

You're a student, aren't you? Anyhow, that's exactly why we like an informal forum, as I already explained on IRC when I got in touch with you a week ago.

Depends on the way you look at it; because well, I'm trying to work here... ;p

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Sat Nov 02, 2013 2:16 pm Post subject:

TomWij wrote:

Yet that doesn't prevent it from happening again.

And that doesn't change that this is not paid development.

Besides, portage has kept going, long past its sell-by; do you really contend that pkgcore, with a vastly improved design (as opposed to ad-hoc piecemeal kludge on top of kludge since "it cannot stop working. ever") will not? And after all, if it's the Gentoo package-manager, the same thing will apply: you can use it in the tree when the EAPI has been approved for use in the tree; which only happens after PM buy-in, which means "can we have this in the next N weeks?" has been answered affirmatively, and only after an implementation is actually up and running if it's non-trivial.

There really is not that much more to add at PM level; the only thing that needs to change on a technical level, but never will, is to stop using sub-slot operators, designed for java and python, to implement LIB_DEPENDS. It's about as sane as EAPI-in-suffix. Both cover the issue: they just do it in a completely cack-handed, and apparently as inconvenient manner, as one could dream of.

Practically an anti-pattern, both of them.

steveL wrote:

Depends on the way you look at it; because well, I'm trying to work here... ;p

Ah more definitions? Good, just so long as we can discount all your talk about deadlines and customers in the real world. Clearly you've just made up a claim to experience you do not possess. Moving on, those considerations do not apply in any case: FLOSS is allowed to grow organically, and as discussed, Gentoo would progress at whatever pace the pkgcore developers and zmedico implemented new features. Not that more major features are needed, at PM level. BASH changes are always simple to add; and what is really needed at PM level is a consolidation phase, and a focus of effort on a next-gen PM that truly addresses all the concerns, with a sane development team.

Just my take on it ofc; no idea if zmedico would enjoy that, or want to take it on. I know he likes working with ferringb's code though, which is why frozen sets (I think they're called) were imported into portage directly from pkgcore, hmm i think it was about 2 years ago.
--

If you're using nvidia-drivers this bug may be apposite (latter is dupe of first, but is the same issue of portage slowing down.)

People in #-portage have also mentioned FEATURES=xattr as an issue.

Anon-E-moose
Watchman

Joined: 23 May 2008
Posts: 6098
Location: Dallas area

Posted: Sat Nov 02, 2013 2:40 pm Post subject:

steveL wrote:

No, that does not mean I am saying incremental development is the only method (before you start shooting down my statements on those grounds): a balance in all things, is required. Incremental development does not mean kludge on top of kludge: it means finishing one thing properly before you go on to the next. And not being afraid to rework what you have literally just written and think is the bees-knees, or throw it away and start again: which requires one not to be too precious about one's work. Without critical self-evaluation, a "developer" is not a programmer. Nor can they ever be.

Agreed, when I was programming, incremental was the only way things could get done,
due a variety of constraints, at many of the places I worked..

I have written many things then turned around and trashed most or sometimes all of what was written because,
user requirements changed, I realized that it wouldn't do what I wanted, it was too complex, etc.

Having a complete programming spec is nice, but I've very seldom seen it in the real world, at least where I worked.

Edit to add: Unfortunately most of the time the ivory world view, ie academic, doesn't match up to what is needed or used in the real world.
It's all well and good to learn to do things in school they way they should be done, but there should be an understanding instilled
in the students that it isn't likely to be that way everywhere.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

TomWij
Retired Dev

Joined: 04 Jul 2012
Posts: 1553

Posted: Sat Nov 02, 2013 3:11 pm Post subject:

steveL wrote:

Besides, portage has kept going, long past its sell-by; do you really contend that pkgcore, with a vastly improved design (as opposed to ad-hoc piecemeal kludge on top of kludge since "it cannot stop working. ever") will not?

That depends on whether or not it will let things happen again; you can look at the past and learn from it, but you cannot contend that the future will go all too well if you just let it go laissez-faire.

steveL wrote:

There really is not that much more to add at PM level;

There however might still be the need to change stuff; as the Portage tree complexity grows over time, which is what we're trying to prepare for in this discussion. I think we're on par here as you suggest that too:

Quote:

to stop using sub-slot operators, designed for java and python, to implement LIB_DEPENDS. It's about as sane as EAPI-in-suffix. Both cover the issue: they just do it in a completely cack-handed, and apparently as inconvenient manner, as one could dream of.

Practically an anti-pattern, both of them.

steveL wrote:

TomWij wrote:

Depends on the way you look at it; because well, I'm trying to work here... ;p

Ah more definitions? Good, just so long as we can discount all your talk about deadlines and customers in the real world. Clearly you've just made up a claim to experience you do not possess.

Experience I have seen and/or learned about; so, well, yeah, to some extent not all of what one learns applies in the real world, but it's up to the reader to choose what to believe and what not to.

steveL wrote:

Moving on, those considerations do not apply in any case: FLOSS is allowed to grow organically, and as discussed, Gentoo would progress at whatever pace the pkgcore developers and zmedico implemented new features.

Due to things like the Portage tree complexity, which is a consequence of the community (including developers) asking for more; these considerations do apply, there isn't progress if the package managers fall short to keep up with the Portage tree complexity. And that either means to improve the package manager, which needs a lot of discussion and work; or to make the Portage tree more complex, which means we go back to the old days of running the long `revdep-rebuild` command the community might be starting to forget about. We've given them shiny rebuilds, I hope we can keep that by improving our package manager(s) such that we do not have to take it back away from them...

steveL wrote:

and what is really needed at PM level is a consolidation phase, and a focus of effort on a next-gen PM that truly addresses all the concerns, with a sane development team.

+1 That or a huge profiling and refactoring session.

steveL wrote:

Just my take on it ofc; no idea if zmedico would enjoy that, or want to take it on. I know he likes working with ferringb's code though, which is why frozen sets (I think they're called) were imported into portage directly from pkgcore, hmm i think it was about 2 years ago.

Not sure about the details; but from what I can tell, zmedico is on a break right now as we haven't seen him doing commits or present in #gentoo-portage for quite some days. He very well deserves that break for all the great work he has done over the years. It would be disrespectful to speculate for whatever reason that is; he might very well be on vacation, or perhaps thinking it through how to move forward. We'll see; but however need to take that into account to keep his wonderful Portage alive, it is after all the heart of Gentoo. If there is one person to be considered as not replaceable in Gentoo, that person would be zmedico.

If Portage gets to be past its time; I don't think he would bother, maybe he even steps up to start helping to get another package manager (like pkgcore) off the ground.

Anyhow; we're not there yet, who knows this entire thread is just speculation and things might as well resolve in another way.

steveL wrote:

If you're using nvidia-drivers this bug may be apposite (latter is dupe of first, but is the same issue of portage slowing down.)

People in #-portage have also mentioned FEATURES=xattr as an issue.

https://bugs.gentoo.org/show_bug.cgi?id=487700#c0 doesn't contain FEATURES=xattr; but well, I've asked in the other bug for emerge --info anyway so we get more data and it might be interesting to see if that one mentions xattr or not.

steveL
Watchman

Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

Posted: Mon Nov 04, 2013 7:34 pm Post subject:

Anon-E-moose wrote:

steveL wrote:

Without critical self-evaluation, a "developer" is not a programmer. Nor can they ever be.

Exactly: in fact, a couple of iterations are always needed, just so you can explore the problem-space. Too many nowadays think that the first thing that roughly works should be pushed to end-users and then "incrementally improved." But they haven't really explored the domain as yet, so end up in a running battle to patch a crap design, instead of getting the design right first.

Quote:

Having a complete programming spec is nice, but I've very seldom seen it in the real world, at least where I worked.

Yup. And as you said the requirements change: again, part of exploring the problem space, with your users.

For a long while there was a tendency to treat the requirements spec as an exhaustive list of everything that had to be delivered: which resulted in spending far too long drawing it up, and then realising that major essential parts, which only showed up during development, are missing. And of course, changing the spec is far too daunting (since it's effectively a contractual negotiation) so what everyone knows is a sucky design gets implemented with an ad-hoc impl of the essentials that went under the radar, and a hope to sort it out "in the next version." But of course, you've wasted a great deal of time by then, so your crap design isn't quite so "state-of-the-art" any more.

Again, a balance needs to be kept: and most of all programmers need to be shielded from office politicking, with a nice quiet space they can focus in, and no distractions. Never happens of course ;)

Quote:

Unfortunately most of the time the ivory world view, ie academic, doesn't match up to what is needed or used in the real world.
It's all well and good to learn to do things in school they way they should be done, but there should be an understanding instilled
in the students that it isn't likely to be that way everywhere.

Agreed. And students should realise that what they have "learnt" about work and deadlines, is nothing on real-world experience where your living, and that of your family, is on the line, not just your pride. Theoretical knowledge of algorithms etc is also needed: but a coder reading about that for the first time, is usually in-love with the whole thing; someone else has been there and done the work for you. It's a bit like diving into Linux for the first time, and finding all this software for free, and getting Gentoo for the first time, and never having to worry about headers or source-code being available: a cornucopia of stuff.

There's an awful lot of nonsense in academia, just like everywhere else, and CS is not what it used to be, by any means.

ArneBab
Guru
Guru

Joined: 24 Jan 2006
Posts: 429
Location: Graben-Neudorf, Germany

Posted: Thu Nov 07, 2013 1:32 pm Post subject:

Note on pkgcore: There are some convenience features of portage which are still missing in pkgcore, and I think that those require quite some effort. They include multiple jobs (--jobs N), quiet display (when emerging multiple packages, only show the build log of failed packages - and only after finishing), and so forth. _________________ Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de ) pkgcore: So fast that it feels unreal - by doing only what is needed.

Dr.Willy
Guru
Guru

Joined: 15 Jul 2007
Posts: 547
Location: NRW, Germany

Posted: Thu Nov 07, 2013 2:53 pm Post subject:

ArneBab wrote:

Note on pkgcore: There are some convenience features of portage which are still missing in pkgcore, and I think that those require quite some effort. They include multiple jobs (--jobs N), quiet display (when emerging multiple packages, only show the build log of failed packages - and only after finishing), and so forth.

make is already parallelizing the most time-consuming part of package installation.

ArneBab
Guru
Guru

Joined: 24 Jan 2006
Posts: 429
Location: Graben-Neudorf, Germany

Posted: Fri Nov 08, 2013 9:44 am Post subject:

Dr.Willy wrote:

ArneBab wrote:

make is already parallelizing the most time-consuming part of package installation.

First on the detail:

It only parallelizes the actual compilation, but not the configuration. Also there are packages whose build process cannot be paralellized very well. In an ideal world, building a package would constantly max out all your cores. In the real world, a good deal of the time is spent with just a few percent load (the display in top).

This does not mean that building with portage is faster, though, because that saved time could well be shorter than the additional dependency calcutalion time.

Now to the general point:

It does not matter whether one given feature provides advantages. The point is that there are many convenience features which are still missing in pkgcore and which will require a substantial amount of work.

But that should not block a pkgcore release. It would just delay a general switch of Gentoo to pkgcore - if the portage devs want such a switch.
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.

yu.cy
n00b
n00b

Joined: 09 Nov 2013
Posts: 3

Posted: Sat Nov 09, 2013 3:30 am Post subject:

Are there any way to make portage to emit the package dependency requirement in a intermediate language(json, xml, etc.), then offload the dependency calculation to a highly optimized external program(or even powerful remote machine) ? So we can just optimize the implementation of this dependency calculation program instead of change portage, and we could also make different interchangeable dependency calculation program to try different dependency resolve strategy. PS: I am not a new Gentoo user, but not familiar with portage system, so please forgive me if I miss something.

mv
Watchman

Joined: 20 Apr 2005
Posts: 6747

Posted: Sat Nov 09, 2013 9:02 am Post subject:

yu.cy wrote:

Are there any way to make portage to emit the package dependency requirement in a intermediate language(json, xml, etc.)

The existing portage tree is probably the only reasonable format unless you want to drop the --autounmask* features.
These are very important to get a chance for at least reasonable error messages, but they require knowledge of practically all data, and it makes no sense to transfere simple formats like /etc/portage/package.{use,mask,umask,accept_keywords} to cumbersome formats, keeping information about the sourcefiles for proper messages.
This is probably the part of the algorithm which needs most backtracking, i.e. when you set all --autounmask* stuff to "no", you will probably get an enormous speed increase, anyway.

yu.cy
n00b
n00b

Joined: 09 Nov 2013
Posts: 3

Posted: Sat Nov 09, 2013 11:39 am Post subject:

mv wrote:

yu.cy wrote:

Are there any way to make portage to emit the package dependency requirement in a intermediate language(json, xml, etc.)

Translate all config file to a standard format(maybe a formal description of the constraints) make external dependency calculation program(let's call it dependency calculation backend, or backend) simple and portable.

And I think there're two part of information need to transfer to the backend, first is the environment, include dependency relation of the packages in the whole portage tree and user configured use, mask, keywords, etc., and second part is current package dependency constraints need to resolve.

The first part can be generate using a mixed strategy, portage related part are generated when sync portage tree(similar to eix), user config part can be translate to intermediate language by portage when emerge command is invoked.

When backend return the result back to portage, whether success, fail to meet some constraint or give some suggestion(drop some constraints), portage should take the charge of interpret the result and translate to user friendly message then output to terminal.

ArneBab
Guru
Guru

Joined: 24 Jan 2006
Posts: 429
Location: Graben-Neudorf, Germany

Posted: Sat Nov 09, 2013 3:41 pm Post subject:

yu.cy wrote:

Maybe you should have a look at the pkgcore code to see whether it already does something like this - but with inside knowledge. For the demanding stuff pkgcore uses the snakeoil library.

But the reason why it is so ridiculously fast is something different: Instead of loading the whole portage tree before doing anything, it only processes the parts it needs to process. You would lose that when you first translate everything to a generic format and hand off all the work to a generic library.
_________________
Being unpolitical means being political without realizing it. - Arne Babenhauserheide ( http://draketo.de )

pkgcore: So fast that it feels unreal - by doing only what is needed.

mv
Watchman

Joined: 20 Apr 2005
Posts: 6747

Posted: Sat Nov 09, 2013 3:51 pm Post subject:

yu.cy wrote:

So instead of the simple and portable

/etc/portage/package.use/russian wrote:

sys-apps/portage linguas_ru

you want to translate it on the run into some "standard" format like

Quote:

<use-flag-change type=user-config where="/etc/portage/package.user/russian" line=1><use=1>linguas_ru</use></use-flag-change>

which in turn is parsed with some bulky library like libxml2. Sounds like something which speed up things enormously and makes writing the resolver a one-liner. Seriously, except for some very few files like /etc/portage/sets.conf practically everything already is in a standard format which is simpler to parse than any other format.
Not to mention that in practice almost never the whole portage tree needs to be parsed for resolving while your suggestion would mean that everytime all information has to be parsed, translated, and parsed again. On systems with 5MB memory say, this alone is a process of some minutes everytime (because the file cache will not be sufficient), and then you have not even started the actual algorithm.

Quote:

And I think there're two part of information need to transfer to the backend, first is the environment, include dependency relation of the packages in the whole portage tree and user configured use, mask, keywords, etc., and second part is current package dependency constraints need to resolve.

You forgot the profile and all the defaults specified there. And remember that you have to keep track for every tiny bit of information where it was from since you have to report it back to the user in case of problems.

Quote:

When backend return the result back to portage, whether success, fail to meet some constraint or give some suggestion(drop some constraints), portage should take the charge of interpret the result and translate to user friendly message then output to terminal.

Ah, so suddenly portage should be able to do the magic of a reasonable outpout without having access to the full tree to get all information.

TomWij
Retired Dev

Joined: 04 Jul 2012
Posts: 1553

Posted: Sat Nov 09, 2013 4:06 pm Post subject:

mv wrote:

So instead of the simple and portable

/etc/portage/package.use/russian wrote:

sys-apps/portage linguas_ru

you want to translate it on the run into some "standard" format like

Quote:

<use-flag-change type=user-config where="/etc/portage/package.user/russian" line=1><use=1>linguas_ru</use></use-flag-change>

Quote:

Yeah, definitely a wrapper that selectively reads data as opposed to converting all data a would work better here; but the question then rather is how good. Now this statement goes for usage cases; but what if the goal of the conversion is to see if some other algorithm can work better than that Portage does without intending to bring out that same program, then the conversion itself to fit the other algorithm doesn't really matter. But then again, what's cheaper to do; reimplement the algorithm to use Portage data, or convert the data to fit the algorithm?

mv wrote:

Quote:

Ah, so suddenly portage should be able to do the magic of a reasonable outpout without having access to the full tree to get all information.

--nodeps can actually do this when you pass it multiple ATOMs; so, reasonable output shouldn't be much of a problem. (And if it were, it's probably one of the less hard thing to fix)

Display posts from previous:

	Gentoo Forums Forum Index Portage & Programming	All times are GMT Goto page Previous 1, 2, 3, 4, 5, 6, 7 Next
Page 6 of 7

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Copyright 2001-2024 Gentoo Foundation, Inc. Designed by Kyle Manna © 2003; Style derived from original subSilver theme. | Hosting by Gossamer Threads Inc. © | Powered by phpBB 2.0.23-gentoo-p11 © 2001, 2002 phpBB Group
Privacy Policy