Any comments on the latest Distrowatch Weekly?

[Kasumi_Ninja]

Another week and another link to an article at Funtoo, a blog maintained by the founder and former chief architect of Gentoo Linux, Daniel Robbins. This time it's all about Portage (the venerable Gentoo package manager) and its increasingly sluggish performance: "One challenge that Portage is facing is that it is essentially trying to achieve several divergent goals - be a ports system for a meta-distribution and also provide a good and safe user experience for Gentoo users. In some cases, Portage can't really do a good job in both areas at the same time. Here's why. As a meta-distribution, Gentoo can have very complex dependency chains. However, as a user-focused distribution, you kind of want the dependency chains in Gentoo to be as straightforward and elegant as possible, without any weird conflicts - in other words, have developers do a lot of the heavy lifting to make dependencies less-fine grained and eliminate strange corner cases and blockers. Yet this hard work impacts the ability of Gentoo developers to keep the Portage tree up-to-date." The story was prompted by the author's tests revealing that an older version of Portage performed better than a more recent one and that the Beagle search engine was partly responsible for the application's sluggishness on the latest release of Sabayon Linux. Read more in this report.

Did they even bother to ask the Gentoo devs opinion? Why put Sabayon and Gentoo on the same heap What are the facts? Is portage that slow (you should check opensuse's package manager)? Why put a opinion piece in the news section?

[Gergan Penkov]

reading drobbins blog and distrowatch, one could see that distrowatch, is just sensationalist piece of crap.
as to enabling beagle per default shows just suicidal tendencies and attempt to ride the suse hype. even ubuntu didn't go with beagle, because of the negative impact this application could throw on them, it is simply dog slow

[wyv3rn]

I am getting really tired of this crap. Gentoo is fine. drobbins seems to think the world has/will stop turning without him. We are all thankful for what he started but for god sake stf crying about leaving your baby to others and then wanting it back/wanting to have a relationship with it when you now feel like being a parent. If portage is so bad write a replacement and show that you can be of value to Gentoo again rather than running your mouth about how Gentoo is just DYING without you as oversear.

My experience is portage has increased in speed greatly since the 2.0.x days, even with added features. IMO its in the best shape it has ever been in. Plenty speed even on some of my older, slower P3 machines.

As for distrowatch.. decent comparison tables, statistics, etc. but their editorials are sensationalist crap to draw readers (drama works) and to get /., other sites to link their articles.

[Kasumi_Ninja]

Good to hear I am not the only who who is fed up with the DW tabloid. I want to write an article myself addressing the 'problems' mentioned in the article. Any factual data for example about portage speed would be greatly appreciated!

[Millzee]

Actually, in all fairness to Robbins, he never really made the claim that Sabayon and Gentoo were the same, just that he was testing out portage and noticed two problems (one on Gentoo and one related to Sabayon's use of Beagle, both of which he maintains a strong interest), there was consulation with developers and it did lead to the problem being identified and solved. Given that a problem was identified and solved in relation to speed, I'd treat it as being the case (though I didn't notice myself) that such a problem existed, unless the developers themselves (zmedico and Ferringb in particular) denied such a problem existed or that he reported it as he states he did. A problem was supposedly identified, reported and solved and therefore all's well. I cannot really comment on Sabayon, since I only really use it as a liveCD, but I respect their developers and they should find a suitable solution to their own problems.

I would argue that strong hostility to him here is unfair. He has made suggestions regarding the aims of Gentoo as to what he feels is a good aim for the OS to have, but it hardly seems the case he wants Gentoo back, various posts he's made over the past few weeks and rejected the notion of taking it back due to things that he believes (correctly) take priority in his life. His philosophical positions regarding Gentoo may might be right or wrong (in light of any given value of 'success', be it innovation, userbase, et cetera), but they are at least offered in the spirit of dialogue. At the very least, this post has presented nothing that any member of the Gentoo community should get pent up over.

Be well all.

[Kasumi_Ninja]

Actually, in all fairness to Robbins, he never really made the claim that Sabayon and Gentoo were the same, just that he was testing out portage and noticed two problems (one on Gentoo and one related to Sabayon's use of Beagle, both of which he maintains a strong interest), there was consulation with developers and it did lead to the problem being identified and solved. Given that a problem was identified and solved in relation to speed, I'd treat it as being the case (though I didn't notice myself) that such a problem existed, unless the developers themselves (zmedico and Ferringb in particular) denied such a problem existed or that he reported it as he states he did. A problem was supposedly identified, reported and solved and therefore all's well. I cannot really comment on Sabayon, since I only really use it as a liveCD, but I respect their developers and they should find a suitable solution to their own problems.

I would argue that strong hostility to him here is unfair. He has made suggestions regarding the aims of Gentoo as to what he feels is a good aim for the OS to have, but it hardly seems the case he wants Gentoo back, various posts he's made over the past few weeks and rejected the notion of taking it back due to things that he believes (correctly) take priority in his life. His philosophical positions regarding Gentoo may might be right or wrong (in light of any given value of 'success', be it innovation, userbase, et cetera), but they are at least offered in the spirit of dialogue. At the very least, this post has presented nothing that any member of the Gentoo community should get pent up over.

Be well all.

I was referring to the Distrowatch article not DR original post. I think DW interpretation of that article is flawed and at best an opinion piece not a news article.

[slonocode]

I think DW interpretation of that article is flawed and at best an opinion piece not a news article.

There is no interpretation in that news article. It simply states that Drobbins wrote a piece in his blog and explains who he is and why he wrote it.

[Millzee]

I was referring to the Distrowatch article not DR original post. I think DW interpretation of that article is flawed and at best an opinion piece not a news article.

I wasn't disagreeing with you, I was referring to wyv3rn's post in this topic. I have no real intent to defend DW's articles (of which I am not a huge fan). Further, I left it open rather than quoting his specific post, since I was concerned that others would end up attacking Robbins on this topic (or, even worse, bringing up the unhelpful Gentoo v. Sabayon bloodshed) rather than either engaging with his position or making the necessary distinction between him and DW.

Be well.

[Suicidal]

I thought the article was very good and well thought out (funtoo)

One thing I really liked about drobbins was his articles, and I would like to see more like this one.

[Millzee]

I think DW interpretation of that article is flawed and at best an opinion piece not a news article.

There is no interpretation in that news article. It simply states that Drobbins wrote a piece in his blog and explains who he is and why he wrote it.

I think the problem was that Robbins wasn't complaining about increasingly slower speeds in Portage (as the DW article reports as the subject of his blog post), but of a minor (and, as reported in Funtoo, fixed) problem in Portage that lead him to muse on a challenge facing its goal implementation. There is skewering of what's really going on and it does make it sound more critical of Portage than it actually is.

Be well.

[rivitir]

I was reading the following blog post and got a little worried:

http://blog.funtoo.org/2007/07/so-can-i ... -back.html

Personally, I love Gentoo. It's my personal favorite distro, not just because it's so well built, but because of the support it has. I would love to see Gentoo to continue to grow, develop, and expand on it's user base. I've heard of other distros having issues and it ultimately caused the distro to become stagnate.

What exactly is the Gentoo foundation planning on doing to make sure that Gentoo continues?

I know Gentoo has had some management issues in the past, what is the foundation doing to resolve their issues?

[Earthwings]

Merged last post here.

[steveL]

So here are the basic steps: Keep the tools focused on doing a specific thing really well. Transition development teams into small independent groups with their own local policies and approaches, so they can better focus on delivering value to a specific target audience the way that only THEY know how. Then create a larger collaborative ecosystem to tie everything together.

Thing is, that's pretty much how Gentoo works now. There are herds like haskell, java, lisp or php, and they tend to have associated overlays where they develop stuff in collaboration with interested users from the "specific target audience" before they push packages out to the main tree.

I agree about keeping tools focussed: far too many portage feature-enhancement bugs are stuff that should be handled by [topic=546828]scripts[/topic].

The guy just needs to get to know Gentoo again afaic.

[enderandrew]

I think DW interpretation of that article is flawed and at best an opinion piece not a news article.

There is no interpretation in that news article. It simply states that Drobbins wrote a piece in his blog and explains who he is and why he wrote it.

++

Why are people getting upset here? drobbins tested out Sabayon, noticed a problem, and posted a fix on his blow. DW mentioned it.

No where here does drobbins throw any negative light on Gentoo or diss Gentoo. When he discussed Portage, he states the obvious that many have stated, but he also says there is still great potential for portage (which I disagree with). I like Gentoo more than anything else out there, but Portage is dog slow. It has gotten faster, but it still resolves dependencies, and looks up packages light-years slower than rival systems like apt-get.

Portage is powerful in the ebuild system, and the use flags, and the concept is great. The portage devs railed for ages how a C or C++ version wouldn't be faster, and when people started making proof-of-concepts that were faster, they just dismissed them. The portage devs also dismissed the notion of a db backend helping, repeatedly insisting it wouldn't help improve performance, even when people did hack a db backend, and proved it worked. The portage devs eventually followed suit, and implemented their own version quietly.

Again, when the community keeps finding ways to improve portage, the devs keep insisting that portage is fine largely as is, and many of them dismiss alternatives, yet from what I hear, some pkgcore code and design has moved into portage.

Frankly, I know portage is arguably the heart of Gentoo, and people are quick to defend it (reasonably so) that doesn't mean we shouldn't be objective in observing its flaws and deciding on how to best address them. In the end, I think either many of the changes/improvements from pkgcore should be migrated to portage, or a completely new system like pauldis should be considered. My money/vote is with the pauldis crowd, but that's just me.

[Genone]

Portage is powerful in the ebuild system, and the use flags, and the concept is great. The portage devs railed for ages how a C or C++ version wouldn't be faster, and when people started making proof-of-concepts that were faster, they just dismissed them. The portage devs also dismissed the notion of a db backend helping, repeatedly insisting it wouldn't help improve performance, even when people did hack a db backend, and proved it worked. The portage devs eventually followed suit, and implemented their own version quietly.

Please provide some evidence on those claims. What prototypes have been dismissed? Of what did we implement our own version quietly?

Again, when the community keeps finding ways to improve portage, the devs keep insisting that portage is fine largely as is, and many of them dismiss alternatives

Again, please provide evidence for those claims.

[enderandrew]

Portage is powerful in the ebuild system, and the use flags, and the concept is great. The portage devs railed for ages how a C or C++ version wouldn't be faster, and when people started making proof-of-concepts that were faster, they just dismissed them. The portage devs also dismissed the notion of a db backend helping, repeatedly insisting it wouldn't help improve performance, even when people did hack a db backend, and proved it worked. The portage devs eventually followed suit, and implemented their own version quietly.

Please provide some evidence on those claims. What prototypes have been dismissed? Of what did we implement our own version quietly?

A simple Google finds some of this stuff in like two seconds.

http://forums.gentoo.org/viewtopic-t-26 ... art-0.html

Here is one example, of where the users added a db backend to portage. Repeatedly we were told in the forums and in chat, that there would be no speedup, that it couldn't be done in portage, that we simply didn't understand how portage worked, etc. We were told what a stupid idea was, and how it would never be officially implemented. And then when the proof of concept showed how much it helped, instead of working with those already working on such a project, a separate and different implementation actually ended up in portage.

Here is someone giving a rather half-ass answer for not implementing the system, even though it still gives a huge speed improvement:

http://gentoo-wiki.com/TIP_speed_up_por ... _NO_cdb.3F

"A new package might fail, and we can't have that!" Any package in your system could fail, and oddly enough when people make the arguement that portage is too dependent on python and other dependencies, the devs brush that off. I can also dig up various threads over and over again where people have suggested portage rewrites, and every time devs come in and suggest there is no possible improvement in a rewrite, and that a version in another language just won't be faster.

So what about those pauldis benchmarks that blow portage away?

[steveL]

Why are people getting upset here? drobbins tested out Sabayon, noticed a problem, and posted a fix on his blow. DW mentioned it.

I'm not. I was really discussing the wider point he made about how Gentoo should go forward: it's been doing exactly that for quite a while, with exactly that structure. Overlays have only been an accepted part of Gentoo for less than 2 years AIUI, so it's no wonder he doesn't know about them.

No where here does drobbins throw any negative light on Gentoo or diss Gentoo.

He has been quite critical of the dev m-l (which I agree with, although the project m-l seems to have sorted that, we'll see over the new year) and also of the Foundation trustees. While it may be fair criticism, the lack of trustee action hasn't really harmed Gentoo AFAICS, and the new SFC setup sounds cool to me.

When he discussed Portage, he states the obvious that many have stated, but he also says there is still great potential for portage (which I disagree with). I like Gentoo more than anything else out there, but Portage is dog slow. It has gotten faster, but it still resolves dependencies, and looks up packages light-years slower than rival systems like apt-get.

Yeah you should see how quick pkgcore is at that.

Portage is powerful in the ebuild system, and the use flags, and the concept is great. The portage devs railed for ages how a C or C++ version wouldn't be faster, and when people started making proof-of-concepts that were faster, they just dismissed them. The portage devs also dismissed the notion of a db backend helping, repeatedly insisting it wouldn't help improve performance, even when people did hack a db backend, and proved it worked. The portage devs eventually followed suit, and implemented their own version quietly.

Well portage still doesn't have a db backend afaict; there's template code in there, and pkgcore has the same (last time i looked.) I have seen that sqllite thread/wiki article which looked quite nice. You probably know more about it though: is that the one you mean?

Again, when the community keeps finding ways to improve portage, the devs keep insisting that portage is fine largely as is, and many of them dismiss alternatives, yet from what I hear, some pkgcore code and design has moved into portage.

Yeah it has, since the pkgcore dev used to be the main portage dev and they're both in the same language, it makes sense. Beyond a certain point, you might as well just switch completely though. No point rewriting portage from scratch- that's what pkgcore is.

Frankly, I know portage is arguably the heart of Gentoo, and people are quick to defend it (reasonably so) that doesn't mean we shouldn't be objective in observing its flaws and deciding on how to best address them. In the end, I think either many of the changes/improvements from pkgcore should be migrated to portage, or a completely new system like pauldis should be considered. My money/vote is with the pauldis crowd, but that's just me.

As you noted changes are migrated. Personally I don't mind using portage at all, and the one pmerge I tried was blazingly fast at dep resolution.
At the end of the day more time is spent compiling the software, and if I wanted binaries, pkgcore has good support too. I have no need to use ciaranm's project, and I'd really rather not. That's just me, though ;)

[Genone]

Portage is powerful in the ebuild system, and the use flags, and the concept is great. The portage devs railed for ages how a C or C++ version wouldn't be faster, and when people started making proof-of-concepts that were faster, they just dismissed them. The portage devs also dismissed the notion of a db backend helping, repeatedly insisting it wouldn't help improve performance, even when people did hack a db backend, and proved it worked. The portage devs eventually followed suit, and implemented their own version quietly.

Please provide some evidence on those claims. What prototypes have been dismissed? Of what did we implement our own version quietly?

A simple Google finds some of this stuff in like two seconds.

http://forums.gentoo.org/viewtopic-t-26 ... art-0.html

Here is one example, of where the users added a db backend to portage. Repeatedly we were told in the forums and in chat, that there would be no speedup, that it couldn't be done in portage, that we simply didn't understand how portage worked, etc. We were told what a stupid idea was, and how it would never be officially implemented. And then when the proof of concept showed how much it helped, instead of working with those already working on such a project, a separate and different implementation actually ended up in portage.

Maybe because the existing module didn't work with the to-be-released portage versions at that time? (which the author was well aware of, see [bug]83371[/bug]). Also I was more interested in references about th claims you made regarding the behavior of portage devs. Who said it couldn't be done? (whatever "it" is, the main problem with all those "db backend" discussions is that people never properly define what they consider to be a "db backend", also see http://dev.gentoo.org/~genone/docs/fosdem-2007-talk.pdf )

Here is someone giving a rather half-ass answer for not implementing the system, even though it still gives a huge speed improvement:

http://gentoo-wiki.com/TIP_speed_up_por ... _NO_cdb.3F

"A new package might fail, and we can't have that!" Any package in your system could fail, and oddly enough when people make the arguement that portage is too dependent on python and other dependencies, the devs brush that off. I can also dig up various threads over and over again where people have suggested portage rewrites, and every time devs come in and suggest there is no possible improvement in a rewrite, and that a version in another language just won't be faster.

About cdb cache plugin:
a) the cdb backend of that module was only available on some archs, which alone is enough reason to not make it a default
b) Nick (carpaski) explicitly said that it might be included at some point
c) Just look at the thread you linked first, you'll see a significant number of people reporting problems caused by the additional dependency (e.g. "cdb module not found")
d) the point of dependencies is to evaluate the cost vs. the benefits we get of them. Python adds one (heavy) dep for the benefit of a nice RAD programming language with a huge standard library (paludis trades it for a direct dependency on gcc and libstc++ as well as some other support libraries). Cdb would have been a minor, but possibly critical dependency, just for somewhat better performance of some operations.

About rewrites:
Noone ever said that a rewrite wouldn't offer any benefits (at least not in the threads I've been involved in). What has been said is that
a) a rewrite won't be faster just because of a different language
b) the cost of a rewrite would be too high for the gain ("cost" doesn't just mean the initial development time)

So what about those pauldis benchmarks that blow portage away?

I don't think anybody has ever said that portage is faster than paludis, so no clue why you come up with that now. After all paludis uses a super-fast DB backend .. oh wait, i doesn't, so I guess that a DB backend isn't the magic solution to performance problems, just like pkgcore shows that a different implmenetation language isn't a magic solution either.

[jonnevers]

<very good insight into portage>

when in the name of everything holy is portage going to show more then just a single masked package that comes up in it's depgraph? why do I need to run emerge 5 times if there are five masked dependencies.

and yes, I did create a patch that listed masked packages in the standard output. Worked really well until portage changed and I wasn't able to keep maintaining the patch.

this is one of my few remaining gripes and it kills me.

[Genone]

<very good insight into portage>

when in the name of everything holy is portage going to show more then just a single masked package that comes up in it's depgraph? why do I need to run emerge 5 times if there are five masked dependencies.

and yes, I did create a patch that listed masked packages in the standard output. Worked really well until portage changed and I wasn't able to keep maintaining the patch.

Do you remember the bug number?
The main problem with the idea is semantics: You're effectively asking portage to anticipate the users response to a 'masked by' message. If the user reacts in a somewhat different way the presented information is likely going to be incorrect, assuming you're just wanting a list of entries for package.*. Showing the relevant parts of the depgraph (unsatisfied nodes, and possible solutions) would probably work, but that information could easily span multiple screens (some kind of selective `emerge --debug --pretend` output).

[omp]

Case A: Not-masked foo depends on masked bar depends on masked baz.
Case B: Not-masked foo depends on masked bar and masked baz.

While not showing baz in case A would make sense, for the assuming user will unmask reason which you mentioned, I don't see why both foo and baz can't be shown in case B. By showing both in case B, you would not be anticipating the user unmasking bar.

I hope the above makes sense; I might not have clearly explained what I meant.

[Genone]

Case A: Not-masked foo depends on masked bar depends on masked baz.
Case B: Not-masked foo depends on masked bar and masked baz.

While not showing baz in case A would make sense, for the assuming user will unmask reason which you mentioned, I don't see why both foo and baz can't be shown in case B. By showing both in case B, you would not be anticipating the user unmasking bar.

I hope the above makes sense; I might not have clearly explained what I meant.

Yeah, in such limited scenarios it might be safe to display both as the possible solutions for bar and baz don't affect each other. But in case A the solutions for baz depend on which solution the user takes for bar (bar-1 depends on baz-1 while bar-2 depends on baz-2, and so on).

[steveL]

<very good insight into portage>

when in the name of everything holy is portage going to show more then just a single masked package that comes up in it's depgraph? why do I need to run emerge 5 times if there are five masked dependencies.

this is one of my few remaining gripes and it kills me.

Just use autounmask (emerge autounmask) I use it all the time (although hopefully I won't be unmasking too many more things.. ;) The last one I did was:
autounmask -n app-portage/eclass-manpages-20070615
..to test an awk thing out. The -n means don't use versions in unmask or keywords files, although you need to tell it the exact package to unmask (to nail the dependencies down I imagine.) We'll definitely be adding support for this to [topic=546828]update[/topic]. It's lovely :-)

[Genone]

We'll definitely be adding support for this to [topic=546828]update[/topic]. It's lovely

Out of curiosity: who is "we"?

[steveL]

Out of curiosity: who is "we"?

Well I work on update with a colleague/friend in our spare time.