| View previous topic :: View next topic |
| Author |
Message |
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
 Posted: Fri Feb 15, 2019 12:42 pm Post subject: Has anyone run "enterprise kernels" on gentoo? |
 |
|
Long story short is that I'm really fed up with all the kernel regressions and want a more stable kernel.
Yesterday I noticed that perf doesn't work anymore. After much debugging I found the broken patch. https://www.spinics.net/lists/stable/msg285687.html
Just a few weeks ago I ran into another regression that could corrupt any KVM VM, including data corruption to disk. I had to restore that VM from backup. https://www.spinics.net/lists/stable/msg282163.html
For both problems there have obviously not been any testing done at all. As long as an upstream bugfix patch applies and builds on an older kernel they are included without further testing. This is now the 5th kernel regression I encountered that was introduced in the "stable" 4.14 series. Here is what the XFS developer Dave Chinner has to say about the current process:
IOWs, if all you're doing is relying on "fixes" tags to determine
what /might/ be needed in a stable kernel.org update, then your
stable backport process is fundamentally broken. You're going to
break things and make stable kernels worse for your users, not
better.
And that's ignoring the elephant in the room. The big difference
between distro backports and upstream stable kernels is the months
of QA and bug fixing spent on the distro backports before any user
gets near them. "stable" kernels might only get a couple of days of
high level integration testing - it's really only enough to smoke
test everything.
So what I'm considering is avoiding kernel.org kernels and using the kernel from some stable enterprise distro. Has anyone done that successfully on gentoo? Redhat kernels would be the main candidate.
https://lwn.net/Articles/734016/
"It takes Red Hat a year to stabilize the kernel chosen for a RHEL release; roughly 300 engineers work on that task, meaning it takes the company 300 person-years to test and harden a kernel." ... "Red Hat has a large testing lab with something like 6000 machines of various sorts all over the world."
But the redhat kernels are designed to run on redhat systems. If the default .config doesn't expose a bug they might not care about it. SELinux is enabled per default and if a problem is mitigated by the SELinux rules they might not prioritize the fix. Redhat's kernel patches are bundled together in one big blob to prevent people from redistributing them. You have to be a redhat subscriber to get the broken out patches. I would be willing to pay for the stabilization work but redhat subscriptions are really expensive. They include stuff like "unlimited number of incidents and a 2-business-day response service level agreement" and I just want the kernel.
Are there any other good stable kernels like SuSE, Ubuntu or even Oracle's "unbreakable enterprise kernel"? |
|
| Back to top |
|
 |
Proinsias
Tux's lil' helper
Joined: 06 Oct 2014
Posts: 133
Location: Scotland
|
| Posted: Fri Feb 15, 2019 4:13 pm Post subject: |
|
|
| Funtoo have debian and rhel6 available. |
|
| Back to top |
|
|
xdarma
l33t
Joined: 08 Dec 2003
Posts: 719
Location: tra veneto e friuli (italy)
|
| Posted: Sat Feb 16, 2019 8:57 am Post subject: Re: Has anyone run "enterprise kernels" on gentoo? |
|
|
| Quote: |
Here is what the XFS developer Dave Chinner has to say about the current process:
[...cut...]
And that's ignoring the elephant in the room. The big difference
between distro backports and upstream stable kernels is the months
of QA and bug fixing spent on the distro backports before any user
gets near them. "stable" kernels might only get a couple of days of
high level integration testing - it's really only enough to smoke
test everything.
|
You can find gentoo-sources on portage.
Stable release is 4.14.83, same longterm stable branch release in kernel.org is 4.14.101.
So, with stable gentoo-sources you have a more tested sources. For sure not perfect, but more stable.
| Quote: |
https://lwn.net/Articles/734016/
"It takes Red Hat a year to stabilize the kernel chosen for a RHEL release; roughly 300 engineers work on that task, meaning it takes the company 300 person-years to test and harden a kernel." ... "Red Hat has a large testing lab with something like 6000 machines of various sorts all over the world."
|
I think that one year is too long time for using a safe kernel. Maybe stable, but not safe, IMO.
And I'm a little doubtful about "fully-bug-free" redhat kernels, as for every piece of software.
_________________
proud user of faKeDE-4.7.3 -> back to windowmaker -> moved to LXQt |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Sat Feb 16, 2019 10:19 am Post subject: |
|
|
I used to run grsec before they closed their source. Now I just stick to LTS releases.
Oracle has their own RHEL variant that uses something they call UEK, which is basically a RHEL kernel with some Oracle special sauce. Not sure if it's free or not, wikipedia would seem to suggest it is.
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
| Posted: Sat Feb 16, 2019 11:40 pm Post subject: Re: Has anyone run "enterprise kernels" on gentoo? |
|
|
I don't really trust that the gentoo devs are maintaining it properly. They seem to be holding back the updates for some time to look for regressions but that will also holds back important fixes. They need to analyze each new commit to make sure users aren't missing out on some important update. Not every important security fix got a fancy name and logo, many don't even have a CVE. Doing that analysis correctly is very difficult and time consuming considering that the upstream developers are actively trying to hide security vulnerabilities.
But considering how many regressions there are in the stable kernels these days users might be better off blindly holding back a kernel that works despite the lack of security updates. I took a look at the 4.4 series, which is the oldest kernel maintained by Greg KH. It's so old it should be rock solid by now but the number of commits per month is slowly increasing over time. Spectre/Meltdown played a part in that and sysbot is finding lots of bugs now but I also think the standard for getting something into the stable tree has decreased. The increased number of commits also results in an increase of regressions.
| xdarma wrote: |
I think that one year is too long time for using a safe kernel. Maybe stable, but not safe, IMO.
And I'm a little doubtful about "fully-bug-free" redhat kernels, as for every piece of software. |
That's one year spent stabilizing a new major kernel, not point releases. Like what they are doing when going from 3.10 to 4.18 in rhel8. The point is that they have 300 engineers doing stabilization using 6000 machines. I don't think any other distribution spend that much effort. No one has claimed that their kernels are "fully-bug-free". |
|
| Back to top |
|
|
xdarma
l33t
Joined: 08 Dec 2003
Posts: 719
Location: tra veneto e friuli (italy)
|
| Posted: Sun Feb 17, 2019 1:51 pm Post subject: Re: Has anyone run "enterprise kernels" on gentoo? |
|
|
| tholin wrote: |
I don't really trust that the gentoo devs are maintaining it properly. They seem to be holding back the updates for some time to look for regressions but that will also holds back important fixes. They need to analyze each new commit to make sure users aren't missing out on some important update. Not every important security fix got a fancy name and logo, many don't even have a CVE.
|
At least, it's another check about the longterm vanilla kernel.
| Quote: | | Doing that analysis correctly is very difficult and time consuming considering that the upstream developers are actively trying to hide security vulnerabilities. |
So, you don't trust on kernel developers?
| Quote: | | But considering how many regressions there are in the stable kernels these days users might be better off blindly holding back a kernel that works despite the lack of security updates. I took a look at the 4.4 series, which is the oldest kernel maintained by Greg KH. It's so old it should be rock solid by now but the number of commits per month is slowly increasing over time. Spectre/Meltdown played a part in that and sysbot is finding lots of bugs now but I also think the standard for getting something into the stable tree has decreased. The increased number of commits also results in an increase of regressions. |
Sure I have a different opinion: I try to avoid older release and "0" sub version.
| Quote: | | xdarma wrote: |
I think that one year is too long time for using a safe kernel. Maybe stable, but not safe, IMO.
And I'm a little doubtful about "fully-bug-free" redhat kernels, as for every piece of software. |
That's one year spent stabilizing a new major kernel, not point releases. Like what they are doing when going from 3.10 to 4.18 in rhel8. The point is that they have 300 engineers doing stabilization using 6000 machines. I don't think any other distribution spend that much effort. |
If you sum up all sub releases of a longterm stable kernel, maybe it's not so much effort. IMO.
| Quote: | | No one has claimed that their kernels are "fully-bug-free". |
But seems you are more confident on redhat developers than any other developers, kernel included. Looks strange to my eyes.
_________________
proud user of faKeDE-4.7.3 -> back to windowmaker -> moved to LXQt |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Sun Feb 17, 2019 2:18 pm Post subject: |
|
|
I've had my own concerns about a new "stable" version pushed out once or twice every week. But I llike even less the idea of IBM (RedHat) pushing their "the patches they are using" to the stable trees.
Do you think RedHat tests against anything other than systemd? |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
| Posted: Sun Feb 17, 2019 10:31 pm Post subject: Re: Has anyone run "enterprise kernels" on gentoo? |
|
|
| xdarma wrote: | | So, you don't trust on kernel developers? |
I don't trust that they inform users when they patch security vulnerabilities. That's because I know they don't. That has been standard practice for a long time.
https://arstechnica.com/information-technology/2013/05/critical-linux-vulnerability-imperils-users-even-after-silent-fix/
The fix to the Linux kernel was published last month. Its documentation did not mention that the code patched a critical vulnerability that could jeopardize the security of organizations running Linux in highly sensitive environments. This lack of security advisories has been standard practice for years among Linus Torvalds and other developers of the Linux kernel—and has occasionally been the subject of intense criticism from some in security circles.
https://lwn.net/Articles/400746/
In addition, the description in the patch isn't terribly forthcoming about the security implications of the bug. That is in keeping with Linus Torvalds's policy of disclosing security bugs via code, but not in the commit message, because he feels that may help "script kiddies" easily exploit the flaw. There have been endless arguments about that policy on linux-kernel, here at LWN, and elsewhere, but Torvalds is quite adamant about his stance. While some are calling it a "silent" security fix—and to some extent it is—it really should not come as much of a surprise.
https://lwn.net/Articles/704231/
There is nothing new about this practice; Linus and others have long had a habit of, at best, neglecting to mention vulnerabilities that have been fixed in released kernels. There are a number of reasons given for operating this way, starting with a general disdain for the "security circus" and the industry that lives on responding to yesterday's vulnerabilities. Every kernel release fixes a great many serious bugs, they say, some of which certainly have security implications that nobody has (publicly) noticed yet. Highlighting specific vulnerabilities only draws attackers' attention to them while glossing over the fact that the only way to get all of the important fixes is to run the latest releases. Security bugs are just bugs, and we fix them like every other bug.
The message from upstream is that every user should always use the latest and greatest kernel blessed by upstream. But I know from experience that doesn't work well.
| xdarma wrote: | | If you sum up all sub releases of a longterm stable kernel, maybe it's not so much effort. IMO. |
It's a massive effort (if you want to do it right). Here is another part of that quote by Dave Chinner:
I've done my fair share of distro kernel maintenance and 500+ patch
backports in the past. Doing backports requires looking at every
patch that isn't in the older kernel, working out if the change is
necessary and then working out all the dependencies that set of
necessary patches requires. It's time consuming, complex, and easy
to screw up, especially if you just blindly rely on "fixes" or
"stable" comments in commits.
https://marc.info/?l=linux-xfs&m=152103080002315&w=2
full post for those who are interested.
| xdarma wrote: | | But seems you are more confident on redhat developers than any other developers, kernel included. Looks strange to my eyes. |
The upstream team and redhat have different priorities. Upstream kernel values a rapid development phase and features, redhat (or their enterprise customers) care more for stability and few unexpected hiccups. |
|
| Back to top |
|
|
xdarma
l33t
Joined: 08 Dec 2003
Posts: 719
Location: tra veneto e friuli (italy)
|
| Posted: Mon Feb 18, 2019 8:15 am Post subject: Re: Has anyone run "enterprise kernels" on gentoo? |
|
|
| tholin wrote: | | xdarma wrote: | | So, you don't trust on kernel developers? |
I don't trust that they inform users when they patch security vulnerabilities. That's because I know they don't. That has been standard practice for a long time. |
Well, maybe behaviour of kernel developers is more tight to "practical" than "hobbistic/educational" environment.
And you can always look at the code, if you need.
Do you knows if redhat publish a full list of his security vulnerabilities?
| Quote: |
The message from upstream is that every user should always use the latest and greatest kernel blessed by upstream. But I know from experience that doesn't work well. |
IIRC, this way has proven to be correct as long term strategy. Maybe not fits all cases.
| Quote: | | xdarma wrote: | | If you sum up all sub releases of a longterm stable kernel, maybe it's not so much effort. IMO. |
It's a massive effort (if you want to do it right). Here is another part of that quote by Dave Chinner: |
I'm sure is a big effort, but diluted on many sub version. And, maybe, it's a reason to keep the pace of newer kernel: less wasted time on retro-patching for kernel developers.
| Quote: | | xdarma wrote: | | But seems you are more confident on redhat developers than any other developers, kernel included. Looks strange to my eyes. |
The upstream team and redhat have different priorities. Upstream kernel values a rapid development phase and features, redhat (or their enterprise customers) care more for stability and few unexpected hiccups. |
I think even kernel developers care a lot about safety and stability and I think linux kernel is a mature piece of software, not a young project that miss lots of features. But's only my opinion, for sure I can be wrong.
Have you already tested a newer kernel? Like 4.19.xx series, for example.
_________________
proud user of faKeDE-4.7.3 -> back to windowmaker -> moved to LXQt |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
| Posted: Mon Feb 18, 2019 8:52 pm Post subject: Re: Has anyone run "enterprise kernels" on gentoo? |
|
|
| xdarma wrote: | | And you can always look at the code, if you need. |
To actually spot the vulnerability fixes you must have good knowledge of the code and the security considerations involved. I definitely don't have that and even if I did the stable trees get about 20 patches per day. I can't spend that time.
https://lwn.net/Articles/769253/
if we are going to tell people that they should be running stable kernels, those people should not need to employ "an army of engineers" to debug those kernels. The stable kernels we are releasing now, she said, are not ready for production use.
I'm just one person. I don't have the needed army of engineers and neither do gentoo.
| xdarma wrote: | | Do you knows if redhat publish a full list of his security vulnerabilities? |
They request CVEs for the vulnerabilities they ship.
https://www.openwall.com/lists/oss-security/2018/09/03/1
Here is a thread from oss-security where Greg KH (upstream stable maintainer) questions why redhat request CVEs for old bugs and Wade Mealing (a redhat stable maintainer) answer. It illustrates how redhat handle vulnerabilities.
| xdarma wrote: | | I think even kernel developers care a lot about safety and stability |
They do care about stability but there is a conflict between stability on one hand and features, performance and rapid development on the other. Upstream prefers the latter.
This is what Thomas Gleixner (x86 maintainer) has to say about that.
https://lwn.net/ml/linux-kernel/alpine.DEB.2.21.1812200022580.1651@nanos.tec.linutronix.de/
This is a problem which I observe increasing over many years.
The feature driven duct tape engineering mode is progressing
massively. Proper root cause analysis has become the exception not the
rule.
In our normal kernel development it's just annoying and eats up review
capacity unnecessarily, but in the face of a timeline or real bugs it's
worse. Aside of wasting time for review rounds, at some point other
people have to just drop everything else and get it fixed.
Even if some people don't want to admit it, the increasing complexity
of the hardware technology and as a consequence the increasing
complexity of the kernel code base makes it mandatory to put
correctness and maintainability first and not to fall for the
featuritis and performance chants which are driving this
industry. We've learned painfully what that causes in the last year. (referring to Meltdown/Spectre)
| xdarma wrote: | | Have you already tested a newer kernel? Like 4.19.xx series, for example. |
It has only been out for about 4 months. I prefer to wait a few more.
I also know I'll encounter a lot of regressions when I upgrade so I want to do it when I have time for that. |
|
| Back to top |
|
|
Syl20
l33t
Joined: 04 Aug 2005
Posts: 619
Location: France
|
| Posted: Wed Feb 20, 2019 4:51 pm Post subject: |
|
|
I understand that you have problems with regressions. Everybody hates regressions. But... There are, and there always will be, regressions. Even with Redhat kernels. That's why pre-production servers exist, in enterprise.
If you really want stability, the best is to keep your systems as consistent as possible. So you should avoid using third-party software, or using software from distro X on your distro Y. Or you _will_ have problems.
And, if you find a regression when you update a kernel :
- reboot on the previous one,
- search if the bug is already issued, do it if not,
- and, if you can't provide a patch, wait until the bug is fixed.
That's it. |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
| Posted: Thu Feb 21, 2019 11:42 am Post subject: |
|
|
| Syl20 wrote: | | But... There are, and there always will be, regressions. |
All software has bugs but that doesn't mean all software is equally buggy. Just claiming that there will always be regressions is missing the point.
| Syl20 wrote: |
If you really want stability, the best is to keep your systems as consistent as possible. So you should avoid using third-party software, or using software from distro X on your distro Y. Or you _will_ have problems. |
And if I don't mix kernels I _will_ also have problems. The goal is to find a solution that minimizes the amount of problems I have, not eliminating them completely. If I had thought mixing kernels and distributions was problem free I wouldn't have created this post. But even though there will be problems with mixing kernels that could still be the setup with fewest problems. I don't know if that's true which is why I asked if anyone have experiences doing that. |
|
| Back to top |
|
|
Syl20
l33t
Joined: 04 Aug 2005
Posts: 619
Location: France
|
| Posted: Thu Feb 21, 2019 5:21 pm Post subject: |
|
|
| tholin wrote: | | All software has bugs but that doesn't mean all software is equally buggy. Just claiming that there will always be regressions is missing the point. |
Then you shoud look for a better kernel than linux, if you think linux is too much buggy. There are alternatives.
I just said, by experience, that adding complexity or weird tricks are never a good way to minimize the number, nor the impact, of bugs.
| Quote: | | The goal is to find a solution that minimizes the amount of problems I have, not eliminating them completely. |
The best ways I know (except that contributing to the project, which is clearly above my abilities) are on my previous post : pre-production servers, and downgrade if there are regressions. I forgot another one : backups.
| Quote: | | If I had thought mixing kernels and distributions was problem free I wouldn't have created this post. But even though there will be problems with mixing kernels that could still be the setup with fewest problems. I don't know if that's true which is why I asked if anyone have experiences doing that. |
Ok, I develop (and if you wonder why Redhat maintainers spend so much time to stabilize their kernels, perhaps there are some answers here) :
- Redhat/CentOS, debian, ubuntu, and so many distros' kernels are provided already compiled, with a toolchain made by and for their distro, not yours ;
- because of that, all that can be added, including lots of stuff you will never need, is enabled by default (and so, more complexity) ;
- including, perhaps, incompatible stuff (do you use systemd, or SElinux, for example ?) ;
- several of these kernels are even not LTS (ubuntu), or are EOL for a loooooong time (Redhat/CentOS), so the distros maintainers have to backport themselves upstream patches (and so, more complexity) ;
- each distro has its own idea of the FHS (which is however a standard), and of what the FHS doesn't say (subdirs, symlinks...) ;
- do you know if your own initrd (if you use one) will run correctly with those kernels, or will you use the other distro's scripts and packages to generate it ? And perhaps you'll need more dependencies, and more other packages (what about the bootloader ?). In this case, after all, why not using the whole distro, rather than some packages ?
There are so many potential failures (thanks Murphy's law) and so much work. Better keep gentoo-sources. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
