Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How do binary distros do their kernel?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 188

PostPosted: Wed Mar 21, 2018 1:05 am    Post subject: How do binary distros do their kernel? Reply with quote

I have been wondering about how binary distros like ubuntu and fedora do their kernel. Do they really enable ALL options? How do only applicable modules load? Does the software probe hardware EVERYTIME it is booted to find appropriate modules to load?(Seems like it already does even with monolithic kernels) How would I go about making a kernel in the same fashion as say ubuntu does but with gentoo? Is there a guide to this kernel fashion?

Couldn't find any answers on my own so I made this topic. I am accustomed to selecting applicable kernel options as built in, I don't use modules except for audio as I have some issues with it being built in.
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 6172
Location: /usr/lib64/lv2

PostPosted: Wed Mar 21, 2018 2:08 am    Post subject: Reply with quote

I think you want genkernel. Make sure to ~arch keyword it.
_________________
Gentoo Studio in open beta. Feedback wanted.
Back to top
View user's profile Send private message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 188

PostPosted: Wed Mar 21, 2018 2:10 am    Post subject: Reply with quote

audiodef wrote:
I think you want genkernel. Make sure to ~arch keyword it.


Sure but the kernel thing is still a mystery to me. Not sure what you meant by arch keywording also, I did add it to package.accept_keywords
Back to top
View user's profile Send private message
Juippisi
Apprentice
Apprentice


Joined: 30 Sep 2005
Posts: 224
Location: /home

PostPosted: Wed Mar 21, 2018 8:00 am    Post subject: Reply with quote

There's debian-sources available in Funtoo's repository. Just a warning, building it requires over 20 GB space so you can be sure it builds every module ;)

Anyway you can study that ebuild and their kernel eclass to do it on Gentoo as well. It uses genkernel.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9793
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Mar 21, 2018 11:41 am    Post subject: Reply with quote

The_Document wrote:
... Not sure what you meant by arch keywording ...
See the Handbook, Mixing Software Branches topic.

As for your other questions:
  1. Do they really enable ALL options? Pretty much: all non-conflictiing ones.
  2. How do only applicable modules load? Does the software probe hardware EVERYTIME it is booted to find appropriate modules to load? Yes
  3. How would I go about making a kernel in the same fashion as say ubuntu does but with gentoo? Is there a guide to this kernel fashion? As audiodef implied, genkernel does this.
- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2372
Location: Illinois, USA

PostPosted: Wed Mar 21, 2018 3:01 pm    Post subject: Re: How do binary distros do their kernel? Reply with quote

The_Document wrote:
How do only applicable modules load?

In earlier versions of OpenRC and eudev, only modules listed in /etc/conf.d/modules were loaded.
Later, all modules were loaded except those listed in /etc/modprobe.d/blacklist.conf
Generally, modules fail to load if the hardware they support isn't present. I would guess that they try to initialize the hardware and exit when that fails.
Distro's like Ubuntu and RedHat build their kernels with a wide range of modules, try to load them all and the relevant modules manage to load.
I recommend only building modules for hardware that you have. If you have say a built-in nic and an add-in nic, build both modules and blacklist one. That way, it's easy to switch if you have trouble with the hardware, just move your plug, edit the blacklist file and reboot.

https://forums.gentoo.org/viewtopic-t-1053466-start-0.html
https://wiki.gentoo.org/wiki/Kernel_Modules
https://wiki.archlinux.org/index.php/OpenRC

You can find more by googling "openrc load all modules".
Back to top
View user's profile Send private message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 188

PostPosted: Wed Mar 21, 2018 9:19 pm    Post subject: Reply with quote

John R. Graham wrote:
As audiodef implied, genkernel does this.


No it does not, I checked its .config and most items are not enabled however some very ridiculous things like isdn, wimax ect are enabled, I should try out debians kernel tool.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9793
Location: Somewhere over Atlanta, Georgia

PostPosted: Wed Mar 21, 2018 11:36 pm    Post subject: Reply with quote

You're right; I misspoke. The kernel can be configured for many purposes, and depending on purpose, many options are on or off. A typical distro sets up the kernel for desktop usage; this is what Genkernel does. But essentially all of the device drivers are set to be built as modules so that the built-in hardware detection can load them if the hardware is present.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 188

PostPosted: Wed Mar 21, 2018 11:40 pm    Post subject: Reply with quote

John R. Graham wrote:
You're right; I misspoke. The kernel can be configured for many purposes, and depending on purpose, many options are on or off. A typical distro sets up the kernel for desktop usage; this is what Genkernel does. But essentially all of the device drivers are set to be built as modules so that the built-in hardware detection can load them if the hardware is present.

- John


Following the genkernel guide from here https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Kernel ended up not having GPU acceleration because no DRM was enabled and AMDGPU wasen't selected also and Im sure even if it was selected, it would have never even loaded the applicable firmware files because
Code:
$genkernel all
does not install firmware by default and even if it were firmware file names would proably still would need to be manually added to kernel config. I recommend dumping genkernel and using whatever Ubuntu is using, because they obviously got it right. It wold make things simpler for users who don't want to configure their kernels. Not enabling hugely important things such as DRM and AMDGPU is inexcusable especially considering this is for desktop use. Even framebuffer drivers were disabled. Genkernel is a broken tool.

Made a bug:
https://bugs.gentoo.org/651126
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2372
Location: Illinois, USA

PostPosted: Thu Mar 22, 2018 2:10 am    Post subject: Reply with quote

The_Document wrote:
and I'm sure even if it was selected, it would have never even loaded the applicable firmware files because
Code:
$genkernel all
does not install firmware by default and even if it were firmware file names would probably still would need to be manually added to kernel config.
I'm willing to be corrected but I was under the impression that if you built a driver as a module, you didn't have to have the firmware in the kernel and if you emerge linux-firmware, the module will find it on it's own. Unless you have an unusual file structure. I've used, on various machines, the radeon and nouveau drivers as modules and never included any special firmware. Likewise the r8169, r8138too and ieee1000 drivers.

Amdgpu is relatively new and might not be supported by genkernel yet.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6487

PostPosted: Thu Mar 22, 2018 10:24 am    Post subject: Reply with quote

they just build the kernel with make allmodconfig mostly and enable some critical ones that "should" appears in most users system and are critical to boot : ie, any generic kernel should have ahci build-in because everyone use such controller, if you want a kernel able to boot most adaptec raid cards, you will buildin aacraid (and this is where ubuntu and friends are bloated, because if user is not using aacraid, it will be load unconditionally as only a module may not be loaded if hardware is not present). For ahci it's like a 99% good bet, and even it might appears little to load aacraid, you have to do that for every controller user "might" have, and the list gets long then, adding little bloat over previous one (which might be lowered thru grub option with some "normal kernel" + "kernel for raid" options and proper initram set).
That's the biggest problem there: what controller you should build-in to get your kernel working fine with most configurations.
You have also usb and at keyboard support, for usb build-in or not, you might bet the computer has some usb ports and it will certainly be use, so build-in usb support allowing to boot an usb keyboard, but at keyboard support is adding bloat for user with usb keyboard...

for the other "non critical" modules to boot, they just let the device manager do the work for them as it will autoload module that are needed for hardware.

for other "software" tools (configs or loop per example), you have to either add them to your init autoload module list, or just use them as build-in (again always loaded even not of any use), but you can leave the task to the user to load them when need.
This gave 3 options: opt-in (user load them), opt-out (you have load them, user have to unload them), or bloat (you don't use them as module but buildin).
I don't know which options they use, i think a mix of opt-out and opt-in, and because some hard dumb, maybe bloat too.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5272
Location: Removed by Neddy

PostPosted: Thu Mar 22, 2018 11:00 am    Post subject: Reply with quote

this is Ubuntu's config

https://gist.github.com/eeyrjmr/279a0930253831a34be53e74ae4311cb
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king


Last edited by Naib on Thu Mar 22, 2018 11:44 am; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 40479
Location: 56N 3W

PostPosted: Thu Mar 22, 2018 11:11 am    Post subject: Reply with quote

Naib,

Postbin please.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 6172
Location: /usr/lib64/lv2

PostPosted: Thu Mar 22, 2018 12:28 pm    Post subject: Reply with quote

If you do a manual kernel config, this tool is very useful. Use it with lspci -n: https://kmuto.jp/debian/hcl/

If you use genkernel, it pays to RTFM. There are many options, including saving/using custom configs to get the desired outcome.

It looks like with a combination of the tool above, genkernel options, and studying what other distros do, you could come up with a custom genkernel config to reliably maintain and update kernels as desired. If you, please share, others might benefit. :)
_________________
Gentoo Studio in open beta. Feedback wanted.
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 6229
Location: Saint Amant, Acadiana

PostPosted: Thu Mar 22, 2018 12:45 pm    Post subject: Reply with quote

Unfortunately kmuto.jp is not maintained. When I drop in my lspci -n it shows no drivers for my 2+ years old motherboard. Use https://cateee.net/lkddb/ instead.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12125

PostPosted: Fri Mar 23, 2018 1:53 am    Post subject: Reply with quote

Tony0945 wrote:
I'm willing to be corrected but I was under the impression that if you built a driver as a module, you didn't have to have the firmware in the kernel and if you emerge linux-firmware, the module will find it on it's own.
That is mostly true. To be precise about the rules:
  • Firmware must be available when the module initializes.
  • Firmware in the standard place on the filesystem can be found automatically, but only if the filesystem is mounted and usable when the module initializes.
  • Modules are normally stored on the filesystem, so they cannot be loaded before the filesystem is ready. This ensures that firmware is also available.
  • Modules can be placed in an initramfs. If so, they can load before the filesystem is ready and will not have access to on-disk firmware. There are some other similarly unusual circumstances where a module can load without the main filesystem available and ready.
That last point is the tricky one. It makes your observation incorrect in a very small number of specialized cases, but your statement is accurate for everyone else.
Back to top
View user's profile Send private message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 188

PostPosted: Fri Mar 23, 2018 1:59 am    Post subject: Reply with quote

Naib wrote:
this is Ubuntu's config

https://gist.github.com/eeyrjmr/279a0930253831a34be53e74ae4311cb


Will try it, hopefully I wont load a bunch of pata drivers. Can anybody tell me what commands are issued to make and install the initramfs?
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 2372
Location: Illinois, USA

PostPosted: Fri Mar 23, 2018 2:49 am    Post subject: Reply with quote

Hu wrote:
It makes your observation incorrect in a very small number of specialized cases, but your statement is accurate for everyone else.

Thank you, that was substantially my understanding but i appreciate your detailed clarification. I don't use an initramfs or a separate /usr and sometimes forget that others do.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 6583
Location: almost Mile High in the USA

PostPosted: Fri Mar 23, 2018 3:19 am    Post subject: Reply with quote

Also keep in mind many of the other distros keep even the rootfs disk driver as a module and loads it upon boot from the initramfs - so no unused drivers remain in memory despite a generic kernel being used.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
The_Document
Apprentice
Apprentice


Joined: 03 Feb 2018
Posts: 188

PostPosted: Fri Mar 23, 2018 6:37 am    Post subject: Reply with quote

audiodef wrote:
If you do a manual kernel config, this tool is very useful. Use it with lspci -n: https://kmuto.jp/debian/hcl/

If you use genkernel, it pays to RTFM. There are many options, including saving/using custom configs to get the desired outcome.

It looks like with a combination of the tool above, genkernel options, and studying what other distros do, you could come up with a custom genkernel config to reliably maintain and update kernels as desired. If you, please share, others might benefit. :)


Pretty useless tool considering its not updated.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum