Portage feature bloat or well needed: namespaces

skellr · Posted: Wed May 29, 2019 8:05 pm Post subject:

Now , I understand why people want to punt the forums...

eccerr0r · Posted: Wed May 29, 2019 11:11 pm Post subject:

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Before you go off on one calling everyone else ignorant and arrogant:

You could've looked up /sandbox in make.conf(5) and disabled it within 2 minutes, but you decided this thread was a better use of everyone's time. You've produced literally nothing to demonstrate these things even cause a slowdown, yet you expect anyone to care.

eccerr0r · Posted: Wed May 29, 2019 11:51 pm Post subject:

Again, the problem is feature creep, and guarantee the workaround/defeature will break at some point. If it's valuable, I'd like to know exactly where the value is, who truly values it.

Else systemd is the best piece of software in the world because it has so many features!
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

skellr · Posted: Thu May 30, 2019 12:47 am Post subject:

Hu · Moderator Joined: 06 Mar 2007 Posts: 21633

eccerr0r · Posted: Thu May 30, 2019 7:48 am Post subject:

Thanks Hu, that was a much better explanation why it was put there. Again I agree that if ebuilds go start using pip or cmake or cargo or wget to download more stuff behind portage's back, then this is a good thing to ensure it doesn't happen.

It was just a bit strange as that ideally once packages are in the tree, this type of trouble has been worked out and the end user does not need to see this - it should be a developer side only thing to ensure QA. Granted since we do sometimes see regular/real filesystem sandbox violations, there's no reason why it could happen for network access "escapes". Thus agreed it's a legitimate reason why these features should exist.

Again I was just caught off guard with the most confusing symptoms for a feature that defaulted on. Normally errors should give hints as to why a particular problem occurred. Both symptoms of the namespace (PID and network) caused very strange portage behaviors that aren't obvious to the underlying problem. Then when finding the true reason behind these errors - seemingly feature creep instead of a perhaps installation error, and the fact it sure seemed more like a developer QA tool (much like sandbox itself) that required kernel options - this was why the feature seemed unnecessary for endusers depending on ebuilds automagically doing the right thing.

If only distcc ended up causing "network sandbox" violations instead of simply giving up on the hosts for no apparent reason... or warning "distcc disabled due to network sandboxing" (prior to the use of automatic proxy) that would have gone a long way to figuring out what was going on...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

krinn · Watchman Joined: 02 May 2003 Posts: 7470

eccerr0r · Posted: Thu May 30, 2019 10:09 pm Post subject:

<sarcasm>Hooray restore points!</sarcasm>
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Hu · Moderator Joined: 06 Mar 2007 Posts: 21633

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

@Hu: Thanks for your comments. I'm neutral on the features, and anything that can help reduce QA burden and possibly have other benefits seems like a net gain.

eccerr0r · Posted: Fri May 31, 2019 5:16 am Post subject:

It's not only having a FEATURE enabled, the FEATURE was stealth enabled without enduser knowledge. It's one thing if an enduser manually enables it and breaks things, it's another when the system stealthily enables it and breaks things. If it had been implemented as a etc-update/dispatch-conf where the edit adds in these FEATURES, that would be acceptable and tip off endusers that something happened and worth at least a web search why the FEATURE is now there.

Agree with the enduser becoming unwitting QA frustrates endusers. Probably beating a dead horse but QA should have been done by the devs/testers, and endusers should not be forced to instrument their systems to become QA by default. Again this is in an ideal world - I do agree that sandbox is a good thing that catches build/install scripts stealthily adding files randomly in your filesystem or download a random file not part of portage, but it's something that in an ideal world, caught by the developers and thus not necessary.

On my "faster"/"high" memory machines I really don't mind doing these QA features, the slower/low memory/customized ones I do. On embedded machines I view CGROUPS as an unnecessary feature and tend to disable all of them to save both code and kernel data that keeps track of these things, and want the minimal, fastest route to distcc to help these machines get their updates built. Perhaps it's only a few kilobytes total, but there are a lot of these little things here and there, both in kernel and in user space.

I will have to say, what's done has been done. My slow machines hopefully will have a continual path to opt out of these QA checks and hope that my faster machines or other people will catch them first... I just hope the path does not evaporate some day.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

asturm · Developer Joined: 05 Apr 2007 Posts: 8936

krinn · Watchman Joined: 02 May 2003 Posts: 7470

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Hu · Moderator Joined: 06 Mar 2007 Posts: 21633

asturm · Developer Joined: 05 Apr 2007 Posts: 8936

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

In complete absence of any factual evidence to the contrary, I assert that this feature *increases* performance.

eccerr0r · Posted: Sat Jun 01, 2019 4:40 pm Post subject:

Hu · Moderator Joined: 06 Mar 2007 Posts: 21633

If your kernel is namespace-enabled, you're paying most of the performance cost whether or not you use the Portage feature at all. In a namespace-aware kernel, all processes are always in a namespace. Pid 1 starts in the initial namespace of each type, and its children are placed there unless moved elsewhere. If you never create extra namespaces, you can pretend you aren't on a namespace aware kernel. There isn't really a check for the process "leaving the namespace" because that's not how this works. Rather, the namespace is a subset of the full system, and anything not visible in the subset simply does not exist, as far as the sandboxed process can see. There's obviously overhead preparing the namespace, but once it is up, the normal rules apply. It's possible that the process in the namespace will be slightly more efficient if the namespace is minimal enough, since there is less work for the kernel to do due to less data with which to work, but that's probably down in the noise for most use cases.

eccerr0r · Posted: Sat Jun 01, 2019 10:33 pm Post subject:

Exactly. Hence the recompile needed to enable namespace/cgroups which clearly adds more text to the kernel and associated data that needs to be tracked if it wasn't enabled in the first place.

I do give benefit of the doubt that these things are done O(1) in the kernel using some hash or using some LUT and not linearly searching, but it's still adding code/data to the kernel that needs to be run/checked, and overall best case would be a wash if it is more efficient, but one had to add text, increasing kernel memory usage.

---

BTW, how does one get distccmon working with the namespaces/cgroups, seems to report nothing now... (perhaps this query should be in its own thread...)
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?