ck-sources : All about choice !

[aCOSwt]

Gentoo is all about choice, so are ck-sources!

Having slept a little, the ck-sources project comes back with new features and new opportunities.

I/ NEW FEATURES

Starting from revision 7 on the 3.5 branch and from the version that will follow 3.4.9 on the 3.4 branch, users will get a couple of opportunities to tune the patching of their kernel via new useflags.

I.1/ bfsonly
Con Kolivas patchsets include the Brain Fuck Scheduler and several other patches designed to increase the system's responsiveness. Of course, any user willing to obtain the lowest possible latency whatever the expenses (i.e., Digital Audio Workstations, Near-To-Real-Time process supervision...) wants the full ck-patchset to be applied and won't set this useflag. This is as the ck-sources always did and this is the default behavior.

This useflag is intended for those who only want to change their default CFS scheduler to BFS. No other patch from the patchset will be applied.

This opportunity might satisfy anyone looking for a general purpose desktop with a better interactivity/responsiveness than what is achievable under a stock kernel but not necessarily sacrificing everything; and in particular, IO throughput, for the sole sake of latency. If all you want is to run some consumer grade multimedia application or game while compiling or rsyncing, try setting this use flag.

I.2/ experimental [From 3.4 to 3.7]
Con Kolivas develops other patches, mainly dedicated to the testing and development of new ideas. This experimental useflag does nothing by itself apart from authorizing the ebuild to take other useflags into account. However, because all the other useflags will actually trigger the commit of highly experimental code (that is to say code that might significantly reduce the performances of the kernel and even break it) we strongly discourage anybody wishing to run a stable kernel to set it. Only users carefully following the status of these patches, involved in their development/testing, or just wanting to have fun should set it.

I.3/ urwlocks [From 3.4 to 3.7]
This useflag will, if the experimental useflag is set, modify the global runqueue of BFS in order to use upgradeable read/write locks instead of the grq spinlock. If you do not know what this means, do not set this useflag.

The current status for this patch is : No demonstrable impact on performances.

I.4/ kvm [3.6]
Kernels of the 3.6 branch will not boot on kvm unless this useflag is set.

Be aware that setting this use flag might cause troubles if the voluntary preemption model is chosen. (CONFIG_PREEMPT_VOLUNTARY=y)

I.5/ deblob
2.6.38-r3 was the last release honoring the deblob useflag. The tree has been made compliant with the freedist license under which the ck-sources are distributed, this enabling any available version to be installed free of... non-free code.

I.6/ In conclusion:

-If you want the ck-sources as usual, want the lowest possible latency whatever the expenses for some near-to-realtime professional purpose: emerge -bfsonly -experimental -urwlocks

-If you want a highly interactive, general purpose desktop running consumer-grade applications: emerge with bfsonly

-If you want to have fun with the bleeding edge, or participate in the development and testing of new concepts, then: emerge with experimental urwlocks

-You intend to boot on kvm? Emerge with kvm!

-You want to free your Gentoo? Emerge with deblob!


II/ NEW RELEASE POLICY


II.1/ No confusing version numbers

All ck-sources packages will be released under the version string used by the gentoo-sources package it is based on.
eg: ck-sources-3.4.9 are based on gentoo-sources-3.4.9

II.2/ Early revision

Some users of the ck-sources are indeed willing to stay on the highest possible version number. For their benefit we will try to go on providing an early release for each new branch. Except in very unusual circumstances, one can reasonably expect the first release of a given branch to be pushed into the tree a week or so after the release of gentoo-sources-V.B.3.

II.3/ Stable-Like revisions

We also want to welcome Gentoo beginners, those of us who wisely maintain their system updated to the last stable version of each package and scrupulously follow the widespread advice not to mix arch and ~arch on their system. For them, we will also do our best to provide on each branch, at least one stable-like version based on the first gentoo-sources stable release available on this branch.

By stable-like, we mean a release that would inherit the qualities of a gentoo-sources stable release, amongst which is the possibility to build it with a stable tool-chain and the aptitude of the resulting kernel to fit perfectly with full stable arches, including in particular the compatibility with stable proprietary drivers. This version will have been previously tested and benchmarked under a stable system.

II.4/ Long Term Support revisions
We also get pro users - users who need their system to work reliably on long term projects and cannot afford the hassle of a kernel version change/qualification for quite long periods of time. For them, we will follow the last long term support branch as far as possible.

II.5/ In short today:

- You want the ck-sources as usual for fun, testing highest possible version numbers even if they are not the best performing kernels? emerge =ck-sources-3.13.5

- You want the last stable-like release? emerge =ck-sources-3.10.32

- You want the long term support branch? emerge =ck-sources-3.10.32

III SPECIAL THANKS TO:

- Markos Chandras,
- thunderrd.

[audiodef]

I'm going to try this out on my laptop.

[aCOSwt]

I'm going to try this out on my laptop.

You are welcome audiodef !
Then I suggest 3.4.9 or 3.5.7 +bfsonly
Go with CONFIG_NO_HZ=y
And do not hesitate to CONFIG_HZ=300 if you need to preserve battery life, results will be better than CONFIG_HZ=1000 with CFS anyway.

And... do not forget : MAKEOPTS="-jX" with X exactly equal to the number of cores.
BFS does not need to be overfed to be efficient.

Anyway... Enjoy and... if the ck-sources can help you augmenting http://audiodef.com/albums.php, I'll be happy !

[audiodef]

Anyway... Enjoy and... if the ck-sources can help you augmenting http://audiodef.com/albums.php, I'll be happy !

I do that on a different machine, using rt-sources. Do you think ck-sources has any latency advantage over rt-sources?

[aCOSwt]

Do you think ck-sources has any latency advantage over rt-sources?

Of course not ! AND Of course yes !

Of course not : Well, these two packages do not really play in the same league.
Thanks to rt-sources, you may expect latencies in the µs magnitude, with ck-sources, you will get overall latencies in the ms range.

Oh wait !

Of course yes : The considerable latency advantage ck-sources get over rt-sources is that... it comes free of extra charges, automagically. You do not get anything to do at all, nothing special to configure, no specifically designed application to run, no specific hardware to have... Make your kernel, install it, boot and fire whatever audio / video app / garland of jack-aware apps... you will obtain an overall latency under the limits of your human capabilities.
Set appropriate priorities, you can even fire a compilation in the background without any impact in terms of Xruns.

Under rt-sources, nothing like this will happen ! If the applications you run under this kernel have not been specifically designed for, that is to say following the art of real-time programming, you are likely to obtain results even worse than under a stock kernel. There are in portage a couple of apps that, if running under an rt-source kernel, will happily transform your 8 cores box into a single task system. Well... I exagerate a little : Into a voluntary preempt system. Well... something like Windows-98...
(BTW : jackd likes real-time priorities but... would indeed get fired into a true real time environment. (Jackd merely passes its time polling file descriptors and polling is evil in a real time context))

For the same reason that polling is something striclty prohibited in true real-time environments, you also need very specific (and generally very expensive) hardware.
All this makes so the rt-sources are more generally found in embedded systems running some process supervision / driving bespoke.

As a conclusion of this, you have a process in which some tasks sometimes need to be granted all the ressources of your system in a couple of µs to the full expense of everything else, then program it and run under rt-sources. In this context, ck-sources would be nothing but a joke. The ck-sources are not a true real-time operating system.

You run on the shelve apps in a desktop environment ? Opt for the ck-sources.
In such a context, rt-sources won't be capable of helping whatever. You'll globally slow down your system (Because the true real time capabilities of the rt-sources imply a significant overhead) and if a single of your apps running is buggy then... you can plug your system off !
(I experienced once some trouble with jackd under the rt-sources... I could not even type kill -9... )

[PaulBredbury]

Note that threadirqs works with a non-RT kernel, these days - worth a try.

[aCOSwt]

Note that threadirqs works with a non-RT kernel, these days - worth a try.

# For a non-RT kernel, use kernel option: threadirqs

You are certainly correct underlining the advantages of threaded irqs but, as far as I understand on an x86 arch, CONFIG_IRQ_FORCED_THREADING is not an option. I mean neither something you can select or not, nor something you need to specifically activate via some boot option.

[PaulBredbury]

Evidence, on 32-bit x86 with PAE:

Code: Select all

$ zgrep THREADING /proc/config.gz 
CONFIG_IRQ_FORCED_THREADING=y

Code: Select all

$ ps -eLo rtprio,cls,pid,pri,nice,cmd | grep "FF" | sort -r | head -n 10
     -  TS 23527  33   0 grep --color=auto FF
    99  FF     8 139   - [rcun/0]
    90  FF    67 130   - [irq/8-rtc0]
    74  FF 26231 114   - [irq/49-nvidia]
    72  FF   276 112   - [irq/46-snd_hda_]
    70  FF    59 110   - [irq/19-uhci_hcd]
    60  FF    64 100   - [irq/1-i8042]
    60  FF    63 100   - [irq/12-i8042]
    52  FF   681  92   - [irq/47-eth0]
    50  FF    61  90   - [irq/18-uhci_hcd]

Those rtprio values are only visible & settable with the threadirqs boot-commandline option.

[audiodef]

If those are indeed the advantage of ck-sources over rt-sources as applied to audio production, I'm definitely going to have to check it out and possibly include it as the recommended kernel for Gentoo Studio instead of rt-sources. I already know that for audio production, low latency in ms in a must, but low latency in µs is technically overkill. With all the overhead you say rt-sources has, ck-sources sounds like a better deal.

Will let you know how it goes when I try it out.

[aCOSwt]

Evidence, on 32-bit x86 with PAE:

Code: Select all

$ zgrep THREADING /proc/config.gz 
CONFIG_IRQ_FORCED_THREADING=y

Well... I did not say that CONFIG_IRQ_FORCED_THREADING does not exist. I did say that it is *not* a selectable option.

Code: Select all

Symbol: IRQ_FORCED_THREADING [=y]
Type  : boolean
Selected by: X86 [=y]

Which means that irq handlers are threaded without any need to select / configure / set anything particular.

Code: Select all

$ ps -eLo rtprio,cls,pid,pri,nice,cmd | grep "FF" | sort -r | head -n 10
     -  TS 23527  33   0 grep --color=auto FF
    99  FF     8 139   - [rcun/0]
...

Those rtprio values are only visible & settable with the threadirqs boot-commandline option.

Under my 3.5.7-ck, I do not set threadirqs in my boot-commandline and I can read

Code: Select all

acoswt@PrimaPratica ~ $ ps -eLo rtprio,cls,pid,pri,nice,cmd | grep "FF" | sort -r | head -n 10
   100  FF    10 140   - [migration/1]
   100  FF     6 140   - [migration/0]
    99  FF  2194 139   - /usr/bin/jackd -P89 -m -dalsa -dhw:3 -r44100 -p256 -n3 -Xseq -P -zs
    99  FF     8 139   - [rcun/0]
    89  FF  2194 129   - /usr/bin/jackd -P89 -m -dalsa -dhw:3 -r44100 -p256 -n3 -Xseq -P -zs
    84  FF  2210 124   - /usr/bin/alsa_in -j cloop -dcloop -q 1
    84  FF  2175 124   - /usr/bin/qjackctl
     1  FF    12  41   - [rcuc/1]
     1  FF     9  41   - [rcub/0]
     1  FF     7  41   - [rcuc/0]

[PaulBredbury]

You're missing the point. Perhaps this is clearer command:

Code: Select all

$ ps -eLo rtprio,cls,pid,pri,nice,cmd | grep "irq/" | sort -r | head -n 10
     -  TS  1973  34   0 grep --color=auto irq/
    90  FF    67 130   - [irq/8-rtc0]
    74  FF  1676 114   - [irq/49-nvidia]
    72  FF   270 112   - [irq/46-snd_hda_]
    70  FF    59 110   - [irq/19-uhci_hcd]
    60  FF    64 100   - [irq/1-i8042]
    60  FF    63 100   - [irq/12-i8042]
    52  FF   633  92   - [irq/47-eth0]
    50  FF    61  90   - [irq/18-uhci_hcd]
    50  FF    60  90   - [irq/16-uhci_hcd]

Those are IRQ handlers, e.g. for video and sound cards, able to have their relative priority changed.

I just rebooted to check (in kernel 3.4.17), and I get the same as you, without the needed threadirqs.

[aCOSwt]

You're missing the point.

I have been missing slightly more than just the point.
It appears I have been unknowingly running with unthreaded handlers...

I just rebooted to check (in kernel 3.4.17), and I get the same as you, without the needed threadirqs.

I just rebooted to check (in kernel 3.5.7), and I get the same as you, with the needed threadirqs...

I stand corrected. Thank you PaulBredbury for this.

[audiodef]

Then I suggest 3.4.9 or 3.5.7 +bfsonly

Doesn't the OP state that for lowest possible latency, one should use -bfsonly?

[aCOSwt]

Well, you had said it was for your laptop so I imagined a more general-purpose use.

To be honest, (when not swapping), the main difference bfsonly vs full ck-patchset is coming from a very simple patch setting the default dirty ratios (these ratios fix some trade-off between throughput and latency)
The higher the dirty ratios, the more applications writing to disk will be privileged and, at least, (because you get a comfortable amount of ram used for buffering) eat all the cpu, which will keep the rest of the applications waiting.

With lower ratios, these applications writing to disk will be slowed down.

So this patch is an intentional attempt to limit the throughput, something desirable in a DAW but not necessarily on a general purpose laptop.

[audiodef]

I have both a desktop DAW and a laptop I use for audio when I need portability, so I'll go with -bfsonly. Thanks for 'splaining.

[audiodef]

I compiled ck-sources and booted up, but...!

My laptop needs broadcom-sta, which needs to be compiled against the current kernel. I did so, and that wasn't a problem, but when the module actually loaded, it crashed the kernel.

This is a problem not specific to ck-sources. It happens to other kernels and so far the only solution I know is to use a lower kernel version... unless you have any ideas for the latest ck-sources and broadcom-sta...?

[aCOSwt]

My laptop needs broadcom-sta, which needs to be compiled against the current kernel. I did so, and that wasn't a problem, but when the module actually loaded, it crashed the kernel.
This is a problem not specific to ck-sources. It happens to other kernels and so far the only solution I know is to use a lower kernel version... unless you have any ideas for the latest ck-sources and broadcom-sta...?

Are you speaking of the troubles met with BCM4313 and the broadcom-sta driver ?
If this is the case then did you test keenblade's solution ? (opensource brcmsmac driver + net-wireless/brcm-firmware)

[Ant P.]

I think I might try this out, since zen-sources seems to be on life support.

[audiodef]

OK, I've tried ck-sources twice on the same desktop machine now. The first time was on an install that had been around for a while. Then for a combination of reasons I decided to backup and do a fresh install of everything.

So now I have a fresh install, and ck-sources freezes at boot with some message about CPU1 and broadcast mode.

What happened?

[aCOSwt]

ck-sources freezes at boot with some message about CPU1 and broadcast mode.

Hrmmm... could you be a little more precise ?
Which version ? which use flags ? which boot parameters ? How many cores do you get ?
Some message about CPU1 and broadcast mode... do you mean this message :

Code: Select all

Switch to broadcast mode on CPU1

?
Is the freeze definitive or does your system wake-up after a while (3-5 minutes) ?

BTW : I had forgot to mention one big advantage of the ck-sources over other *-sources is that ck-sources are a Gentoo supported kernel !
Which, amongst other things means that... posting bugs on gentoo-bugzilla is welcomed.

[audiodef]

ck-sources freezes at boot with some message about CPU1 and broadcast mode.

Hrmmm... could you be a little more precise ?
Which version ?

3.6.2

which use flags ?

None enabled.

which boot parameters ?

Straight up kernel /boot/kern.ck root=/dev/sdc3

How many cores do you get ?

6

Some message about CPU1 and broadcast mode... do you mean this message :

Code: Select all

Switch to broadcast mode on CPU1

?

That's the one!

Is the freeze definitive or does your system wake-up after a while (3-5 minutes) ?

I didn't wait that long, but definitely nothing was happening after a couple of minutes. I'll let you know what happens when I get a chance to boot into ck-sources and let it sit there for 10 minutes.

ck-sources freezes at boot with some message about CPU1 and broadcast mode.

Hrmmm... could you be a little more precise ?
Which version ?

3.6.2

which use flags ?

None enabled.

which boot parameters ?

Straight up kernel /boot/kern.ck root=/dev/sdc3

How many cores do you get ?

6

Some message about CPU1 and broadcast mode... do you mean this message :

Code: Select all

Switch to broadcast mode on CPU1

?

That's the one!

Is the freeze definitive or does your system wake-up after a while (3-5 minutes) ?

I didn't wait that long, but definitely nothing was happening after a couple of minutes. I'll let you know what happens when I get a chance to boot into ck-sources and let it sit there for 10 minutes.

BTW : I had forgot to mention one big advantage of the ck-sources over other *-sources is that ck-sources are a Gentoo supported kernel !
Which, amongst other things means that... posting bugs on gentoo-bugzilla is welcomed.

Nice!

[audiodef]

Well, I rebooted and left it alone. Came back well after 10 minutes had passed and it was still hung on "Switch to broadcast mode on CPU1".

[aCOSwt]

Well, please pastebin your .config and try 3.4.9 or 3.5.7

[audiodef]

Here's the pastebin: http://pastebin.com/QdJwqCR3

Will try an older version and let you know what happens.

[aCOSwt]

Right ! While I cannot tell for sure that these are responsible for the trouble you are experiencing, there are a couple of settings I strongly disagree with (For the sake of coherency with ck-patches) and others that I would suggest to be set differently :
---------------------------------------------
1/

CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT is not set

That is almost certainly *not* what you want.
Location:
-> Processor type and features
-> Preemption Model
You definitely want to select the Preemptible Kernel (Low Latency Desktop) option.

2/

# CONFIG_HOTPLUG_CPU is not set

With your hexacore, that is almost certainly *not* what you want.
Location:
-> Processor type and features
You definitely want to select the Support for hot-pluggable CPUs option.

3/

CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"

With the ck patches, best performances are with the noop io scheduler. Some like the deadline but definitely not the cfq which only adds a non negligible overhead.
Location:
-> Enable the block layer
-> IO Schedulers

This being said, I would vote for n°2 as the most probable culprit.