Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
The quest for the ultimate .config
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Chat
View previous topic :: View next topic  
Author Message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sat Nov 03, 2012 11:51 pm    Post subject: The quest for the ultimate .config Reply with quote

Hi,

this topic may also fit into the Kernel category, but I think it fits here even more, as it's more about sharing opinions and experiences. Mods may observe and move it later if appropriate.

I'm sure I'm not the only one who has (successfully) configured and compiled several kernels, but after some time has become tired of adding modules to the .config as he encounters various hardware.
Sure, usually it's just
Code:

cd /usr/src/linux
make nconfig    #and find the relevant driver and hope it can be configured as a module without rebuilding the kernel
make modules
make modules_install

but sometimes that doesn't work or I need to test some hardware ASAP and telling someone "Give me a sec, I'll configure my kernel" or "Hmm, I cannot tell whether it's faulty or not, I'd have to find, configure and compile the driver first".
AFAIK if I need a new module and must enable some bistate option to get the module tristate option, I have to rebuild the kernel right? Usually it wouldn't compile otherwise.

Don't get me wrong, I don't have the *buntu mentality and I DO care about having my system as lean as possible, however, the environment I'm in forces me to have a more versatile kernel, almost a distro-level kernel.

So, I was thinking about how to configure a kernel which would be almost as sane and lean as a Pappy's kernel seed and yet would have most drivers for most hardware as modules so I wouldn't have to recompile it each time I encounter some new hardware.

Well, here are the most obvious approaches:

  1. make a .config which has most hardware options enabled in such a way, that the relevant driver can be configured as a module on-demand. E.g. if you find out you need some special broadcom wifi card module, you should have the relevant scsi option enabled beforehand
  2. use something as crazy as allmodconfig and try to turn off at least the most useless stuff
  3. start with a kernel seed or my own custom .config and use common sense to enable the most possible modules

For some time I've been using the 3. approach, but even then I always encounter such hardware that I didn't anticipate. The 2. option is quite close but I fear such a kernel would be terribly fat and slow, possibly prone to errors and unwanted enterprise policy/product options restricting normal desktop behavior. The 1. option is quite hard to achieve, because you may not know which drivers require what option and you don't want all drivers too (just for hardware mainly).

So today I've been playing with the idea I've come up with, fusing the 3 approaches together using join, comm, diff, sort to compare and merge .config files generated by allmodconfig, allnoconfig and pappy's kernel seeds, but it didn't work out, because make oldconfig just restarts the whole config if there are too many unmet dependencies.

I've found kconfglib.py, will try to use that instead:

  1. start with a sane base config, e.g. pappy's kernel seed
  2. enable everything tristate as module and enable their respective reverse deps
  3. disable everything tristate, leaving the options enabling their existence as tristate enabled

I've almost managed to do this, but oldconfig wouldn't check all the reverse deps

Now, let me know what you think :)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32099
Location: 56N 3W

PostPosted: Sun Nov 04, 2012 12:02 am    Post subject: Reply with quote

smartass,

A kernel with lots of modules isn't slow and bloated if unwanted modules are not loaded.
It takes a long time to build and it take s more disk space.

Start with allmodconfig then turn on <*> everything needed to boot, so you don't need an initramfs and all the rubbish that goes with it.
Now blacklist anything that loads that you don't want. You can still load it by hand.

You must never edit the .config file bu hand, or with anything except the provided tools as maky config options flip several CONFIG_ flags.
Make oldconfig cannot sort out the mess, so you get a kernel full of strange bugs.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Nov 04, 2012 8:44 am    Post subject: Reply with quote

Thanks for the feedback Neddy.

I've already tried what you described, but I soon realized, that many modules require many other options aimed at specific enterprise platforms.
The inconvenient thing about allmodconfig is that its' too close to allyesconfig. When I make allmodconfig and then find an option which I want to disable (e.g. something for embedded systems or servers), then I must first find all the modules that depend on it. Not impossible, but neither easy.

Although the idea with module blacklisting sounds interesting, it would still require a lot of human intervention.

Well, I got very close to starting at point 2. and getting to point 3., but now I'm wondering how to merge it with the sane base without disabling all the options necessary for later module options. E.g. pappy's config disables many things that I'd like to remain possible.
Guess I'll just have to do it by hand, I have to make some modules as built-in anyways.
Code:
# start with total bloat
make allmodconfig
# disable all modules
sed -i 's/=m/=n/g' .config
# validate and repair the config. This results in a config capable of enabling modules later
# could also use `yes n` as it asks about stuff I don't need (AppArmor, SElinux, TOMOYO,..)
# or just do it manually, as it's not very long
yes | make oldconfig
# set the sane base by hand, build in necessary module-capable stuff
# possibly could have the sane config opened in another window as KCONFIG_CONFIG=config.base make nconfig
# this step could be also done after make allmodconfig instead
make nconfig


This would work. My question is, what's users' experiences with adding modules to the kernel later on vs. really compiling all modules in the beginning?

If that turned out out be a bad idea, I'd think of doing something like
Code:
# start with total bloat
make allmodconfig
#record what was set as module
sed -n '/=m/p' .config |sort -t = > config.mod_only
# disable all modules
sed -i 's/=m/=n/g' .config
# validate and repair the config. This results in a config capable of enabling modules later
# could also use `yes n` as it asks about stuff I don't need (AppArmor, SElinux, TOMOYO,..)
# or just do it manually, as it's not very long
yes | make oldconfig
#now it's easy to disable really unnecessary options and set the sane base
make nconfig
#will work with it later
sed -e 's/# \(.*\) is not set/\1=n/g' -e '/#.*\|^$/d' .config |sort -t = > config.allmod_capable
#enable again all modules
join -v 1 -j 1 -t = config.allmod_capable config.mod_only | sort -t = - config.mod_only > .config
#fix and validate the config
make oldconfig

The last step still restarts the whole config, trying to get my head around it.... Might have to use that kconfiglib.py

EDIT:
Found out this actually works, but feels like a big hack:
Code:
# start with total bloat
make allmodconfig
# record what was set as module
sed -n '/=m/p' .config |sort -t = > config.mod_only
# disable all modules
# it's important that the symbols are completely removed and oldconfig is not called
sed -i '/=m/d' .config
# now it's easy to disable really unnecessary options and set the sane base
make nconfig
mv .config config.base
# add the modules back
cat config.mod_only >> config.base
# make a new config enabling all modules but respecting our disabled options
KCONFIG_ALLCONFIG="config.base" make allmodconfig
# may be necessary to review the .config, as not all disabled options may have been respected
make nconfig

The last step spits out many errors, but works as expected in most cases and is a valid config.
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1160

PostPosted: Sun Nov 04, 2012 10:49 am    Post subject: Reply with quote

cd /usr/src/linux
make help |less

shows:
localmodconfig - Update current config disabling modules not loaded
localyesconfig - Update current config converting local mods to core
_________________
fun2gen2
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Nov 04, 2012 12:13 pm    Post subject: Reply with quote

ulenrich wrote:
cd /usr/src/linux
make help |less

shows:
localmodconfig - Update current config disabling modules not loaded
localyesconfig - Update current config converting local mods to core

Hi,
I am aware of these options, but as I said, I cannot know beforehand what driver I may need tomorrow and I'd like to be prepared for most HW.
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1160

PostPosted: Sun Nov 04, 2012 1:25 pm    Post subject: Reply with quote

Uuuh, excuse me, I didn't read your starting article :(
... because I don't understand what you want (when tried to read it).

I do compile my little vmlinuz faster than a fat kernel. Just ten minutes to go on a tmpfs. It is a throw away commodity for me ... if updated external modules need to be emerged.
_________________
fun2gen2
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Mon Nov 05, 2012 6:34 pm    Post subject: Reply with quote

ulenrich, I'm not exactly sure what I'm looking for, that's why I started this thread to gather experiences and opinions of other users, e.g. compiling all modules vs. adding them in later.

I'm hoping someone with similar trouble will point me in the right direction
Back to top
View user's profile Send private message
augury
l33t
l33t


Joined: 22 May 2004
Posts: 718
Location: philadelphia

PostPosted: Wed Nov 07, 2012 11:49 pm    Post subject: Reply with quote

smartass wrote:

Hi,
I am aware of these options, but as I said, I cannot know beforehand what driver I may need tomorrow and I'd like to be prepared for most HW.


Sometimes you build 'em all. Sometimes you don't. < > Dallas's 1-wire support ---> is not tomorrows option IMO.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Thu Nov 08, 2012 7:27 am    Post subject: Reply with quote

Yes, some options seem more unlikely than others, but then I'm still left just with my best bet.
What I'm asking about is whether people usually just include most drivers as modules, excluding the really strange ones OR whether it's a good idea to build a kernel with most modules disabled, but with the possibility to build them later.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 32099
Location: 56N 3W

PostPosted: Thu Nov 08, 2012 7:27 pm    Post subject: Reply with quote

smartass,

My kernels contain everything I use every day built in. Thats not much more than items needed to boot without and initramfs and the network card.
Eveything else I use occasionally is <M>.

I take pot luck about building modules for new toys without doing a full kernel rebuild and try <M> first.
If it works - fine, if not you usually know before you power down to install the new toy, so very little time is lot in a kernel rebuild.
I assume you don't start from make clean.

Further, I upgrade my kernel every change in x where kernels are numbered 3.x.y.
That means when I try out new hardware I am often due a kernel update anyway.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Dec 02, 2012 4:31 pm    Post subject: Reply with quote

Spent almost the whole day playing with kconfiglib.py, didn't get very far, because the API is !@$#* inconsistent and mostly hidden from the user so getting only the specific kind of a dependency is a lot harder than I thought.

I've seen some Kconfig->XML converter, didn't work. Seems I need something like that, because kconfiglib.py didn't give me direct access to the dependency system.

Anyone know of some other working Kconfig API? At the worst I'll dive into C...

Neddy: Sometimes you just need to test something on the spot and can't restart the computer, because you have other important things connected to or running on it. Otherwise you're right.
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2717
Location: The Peanut Gallery

PostPosted: Sun Dec 02, 2012 7:07 pm    Post subject: Reply with quote

smartass wrote:
I've seen some Kconfig->XML converter, didn't work. Seems I need something like that, because kconfiglib.py didn't give me direct access to the dependency system.

Anyone know of some other working Kconfig API? At the worst I'll dive into C...

I was going to suggest this. It'd actually be quite an interesting thing to do.
Quote:
Sometimes you just need to test something on the spot and can't restart the computer, because you have other important things connected to or running on it. Otherwise you're right.

This seems odd. If it's work situation, why can't you just set aside a machine for hardware testing? I'm not saying you could never use it for anything else, but it should be understood that anyone can get kicked off it, and it'll likely be rebooted to try out something.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Dec 02, 2012 7:20 pm    Post subject: Reply with quote

Hi Steve,
What were you going to suggest? Diving into C or some XML frontend? I've had a look at the C files and it's not very intuitive, can't find any API-like docs.

I wish I had an always accessible testing machine...well, I don't, so I have to find a different way.
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2717
Location: The Peanut Gallery

PostPosted: Wed Dec 05, 2012 9:21 am    Post subject: Reply with quote

smartass wrote:
Hi Steve,
What were you going to suggest? Diving into C or some XML frontend? I've had a look at the C files and it's not very intuitive, can't find any API-like docs.

Yeah, the C stuff. I haven't looked at the code at all, yet, but I'm doing some work on make and build stuff, as well as C programming, so it might be interesting from that perspective, though I'm not sure how much time I would have.
Quote:
I wish I had an always accessible testing machine...well, I don't, so I have to find a different way.

Why not try with a virtual machine? I've had good results with vmware on gentoo (you just have to download a free serial) even on low-powered hardware, though that was a while ago.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Wed Dec 05, 2012 10:33 am    Post subject: Reply with quote

Well, AFAIK from looking at the sources the main kconfig routines all spawn from the lkc.h header, using expr.c, util.c, symbol.c. But they lack any consistent docs, or I haven't looked in the right places.

I'm ok with working in C, but only if the API is consistent, hunting down structs and functions in a bad C API is a nightmare. And so far it doesn't look like a consistent one.
Back to top
View user's profile Send private message
alkan
Guru
Guru


Joined: 06 Aug 2004
Posts: 385
Location: kasimlar yaylasi

PostPosted: Wed Dec 05, 2012 5:24 pm    Post subject: Reply with quote

Here what i've been doing with kernel config.
It served me well over the years on different hardware.

For once manually: set everything as "module" and as "yes" if it is not a moduleable option (I admit it takes a lot of effort)
For a new kernel: start from the old config, apply the first step for the new features (this is minimal from one version to the next).

It takes a lot of effort for initial configuration and then a lot of compile time. But it works across machines and across kernel versions. Changes needed are usually minimal (like CPU option with different machines). I've been maintaining my kernels like this since 2004.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Wed Dec 05, 2012 6:05 pm    Post subject: Reply with quote

alkan wrote:

For once manually: set everything as "module" and as "yes" if it is not a moduleable option (I admit it takes a lot of effort).

Are you aware of
Code:
make allmodconfig
? Sounds like it does what you want automatically.
Back to top
View user's profile Send private message
fpemud
Apprentice
Apprentice


Joined: 15 Feb 2012
Posts: 167

PostPosted: Thu Dec 06, 2012 3:57 am    Post subject: Reply with quote

I have encountered the same problem and I write a small project fpemud-kernelmanager.

In this project I encapsulate the Kconfiglib.py to a neater API set by which I can select the .config items like I do in make menuconfig. It use user defined rule files to select the .config item automatically (in cfg/rules.d).

This project is in initial state, it only supports .config checking. But it already helped me find some errors in my .config.
I managed to get in touch with the author of Kconfiglib.py who is a very nice and active guy and he is trying to improve it.
I'm learning the markdown language of github, after this I will write a README to fpemud-kernelmanager.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Thu Dec 06, 2012 5:37 am    Post subject: Reply with quote

Hi fpemud, didn't know you're on Gentoo :)
I noticed your fork and active discussion on Github.
I myself engaged in discussion with ulfalizer, but as he put it, the API is not capable of what I'm asking for right now. As far as I can see, to support full dependency evaluations the core of the API would have to be redone and ulfalizer is a very busy man.

I think I'll give Cocxml another go, but it fails, it appears to be too strict when it comes to whitespace.

I think I'll look for some lexical analysis tool for Bison or Flex (that's how Kconfig is defined) in Python, PyBison or FBModule look promising.
Back to top
View user's profile Send private message
fpemud
Apprentice
Apprentice


Joined: 15 Feb 2012
Posts: 167

PostPosted: Thu Dec 06, 2012 10:47 am    Post subject: Reply with quote

nice to meet you, smartass :)

Yeah, I'm also having one point unimplemented yet, really unsure how to name it...
I call it "selects calculation", I think it should be the same thing as "dependency evaluations".
Wish I can better describe it in my README.

But to me it's not on the key path, so I can still do many things without it.

I'd like to learn your discussion with ulfalizer, can I see it on web?
I think I still need a better understanding on Kconfig syntax and structure.

Willing to see your work, good luck, buddy.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Thu Dec 06, 2012 12:38 pm    Post subject: Reply with quote

There was no real discussion with ulfalizer, I just asked what other dependency access there is besides Config.dep[Symbol] and Symbol._get_dependent().

If you want to know more about kconfiglib.py, dive into the source, there are many private methods that don't show up in the docs. You'll also find that instances have many attributes, which can be useful. Sadly, most of this stuff is not well documented and isn't very intuitive.

Dependency evaluations is somethign I'm looking for too.

Good luck :)
Back to top
View user's profile Send private message
alkan
Guru
Guru


Joined: 06 Aug 2004
Posts: 385
Location: kasimlar yaylasi

PostPosted: Fri Dec 07, 2012 6:17 am    Post subject: Reply with quote

smartass wrote:
alkan wrote:

For once manually: set everything as "module" and as "yes" if it is not a moduleable option (I admit it takes a lot of effort).

Are you aware of
Code:
make allmodconfig
? Sounds like it does what you want automatically.

I am aware, I tried at some time in the past, I can't remember the specifics but it didn't generate a usable config. And i couldn't boot into it. I guess it is not all that intelligent.
Back to top
View user's profile Send private message
Ulfalizer
n00b
n00b


Joined: 09 Dec 2012
Posts: 3

PostPosted: Sun Dec 09, 2012 1:15 am    Post subject: Reply with quote

I'm the author of Kconfiglib (https://github.com/ulfalizer/Kconfiglib) and came across this while Googling. :)

You're right in that the internal dependency APIs probably aren't very useful. They work the way that is handiest for the implementation, without any concern for what the user might find useful. That's why they're internal though! If there's some functionality that would make your life easier, just suggest it and I might implement it :). Consider that defining what e.g. "A requires B" means might be tricky with cases such as "depends on FOO || BAR" though.

I was thinking of adding some APIs for getting expressions, seeing what symbols are in them, walking over them, etc. That would be the most general thing you could do.

Oh, and if some part of the code seems especially cryptic, tell me and I could add some more comments. I consider Kconfiglib pretty clean compared to the scripts/kconfig/*conf utilities, and some of the mess might be because parts of Kconfig are a mess. :P

Edit: I cleaned up the installation to be minimally invasive. No files need copying anywhere now, and the comments that were added to various files previously were a bit silly; if you've installed the lib, you already know where to find the docs. :P


Last edited by Ulfalizer on Sun Dec 09, 2012 2:09 am; edited 3 times in total
Back to top
View user's profile Send private message
Ulfalizer
n00b
n00b


Joined: 09 Dec 2012
Posts: 3

PostPosted: Sun Dec 09, 2012 1:23 am    Post subject: Reply with quote

Suggestions for improvements to the existing APIs are welcome too of course.
Back to top
View user's profile Send private message
smartass
Apprentice
Apprentice


Joined: 04 Jul 2011
Posts: 189
Location: right behind you ... (you did turn around, didn't you?)

PostPosted: Sun Dec 09, 2012 9:22 am    Post subject: Reply with quote

Damn this world is small :D

Ulfalizer, no offense, but your API isn't well suited for dependency evaluations, that's all.

If you'd want it too, you'd most likely have to reimplement the core to support having tokens as instances and what not.

But that may not be worth it, because you'd be reimplementing the wheel, since there's already the flex/bison Kconfig parser. So maybe it would be better, if we tried to reimplement kconfiglib.py on top of something like pyBison, because then me and fpemud could get what we want and you could get an API that would be faster as it could be based on a C library. Also, if it used some of the C headers for parser definitions, you could be sure it is be compatible with the C implementation of the Kconfig version at hand.
And if it was implemented in such a way, maybe people would consider integrating it into the mainline kernel tree, as it would have a lot more potential with a an API exposing the Bison parsing.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Chat All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum