Benchmarks: Beyond-Sources vs No-Sources.

Message

protex · Post by **protex** » Tue Apr 04, 2006 12:47 am

Does someone want to run some benchmarks between these two kernels, or at least direct me to some software that will? Thank you!

bollucks · Post by **bollucks** » Tue Apr 04, 2006 1:07 am

interactivity:
http://www.kernel.org/pub/linux/kernel/ ... nterbench/
throughput:
http://www.kernel.org/pub/linux/kernel/ ... kernbench/

Tiger683 · Post by **Tiger683** » Tue Apr 04, 2006 1:58 pm

If this is only about who is ricing up the most, then beyond doesn't want to be the winner....

bollucks · Post by **bollucks** » Tue Apr 04, 2006 3:19 pm

Tiger683 wrote:If this is only about who is ricing up the most, then beyond doesn't want to be the winner....

Yeah but what if you just happen to have the longest grains of rice?

nbvcxz · Post by **nbvcxz** » Tue Apr 04, 2006 5:38 pm

protex please let us know what are results of the test; wonder if there are any significant differences

predatorfreak · Post by **predatorfreak** » Tue Apr 04, 2006 5:46 pm

nbvcxz wrote:protex please let us know what are results of the test; wonder if there are any significant differences 8O

The biggest difference between the two patchset's is that -no uses plugsched and -beyond uses straight staircase.
It's also unfair to compare two different CPU scheduler's against one and other, however, you CAN do it. I don't think the difference if you compared -no using staircase and -beyond using staircase would be more than a very small percentage (read: unusable).

nbvcxz · Post by **nbvcxz** » Tue Apr 04, 2006 6:00 pm

Well, I know the difference in builds, but I also wonder if there any difference in overall performance - so far I use sources with Tiger683 patches (great job and very stable for me) and I don't know if there any reason to try NO sources (except of curiosity

)

Tiger683 · Post by **Tiger683** » Tue Apr 04, 2006 6:18 pm

nbvcxz wrote:Well, I know the difference in builds, but I also wonder if there any difference in overall performance - so far I use sources with Tiger683 patches (great job and very stable for me) and I don't know if there any reason to try NO sources (except of curiosity )

curiosity is mother of creativity, nothing wrong with it

but as predatorfreak said: the result can be VERY subjective.

cheater1034 · Post by **cheater1034** » Tue Apr 04, 2006 8:59 pm

You could run a benchmark between the 2 kernels, leaving schedulers at their default (nicksched for no, and staircase for beyond) just to see which scheduler performs better. You could also set staircase for no and see which performs better.

Using the default (nicksched) in no is a perfect way to benchmark the 2 patchsets. Because nicksched doesn't come in -ck, only comes from plugsched (now, you could edit ck to integrate plugsched, but it isn't the wisest of plans because you could just patch up plugsched then the ck patches) Since -beyond does not offer plugsched, it only offers staircase (ck) there will be interactivity, etc, changes depending on what scheduler you use.
I agree with what tiger and predatorfreak said, but since beyond doesn't offer plugsched, you probally will see performance changes (for the better or the worse) if you stick with the -no default.

however, -beyond and -no do differ greatly.

-no is based off of the -mm kernel patchset, (more experimental, tons of patches) , and it uses plugsched (nicksched default) for its cpu scheduler. -beyond is based off of -ck, and uses staircase.

so, -no includes more features, because it is based off of -mm, but -beyond has the edge on stability (not that no doesn't work, it works rather well, it just has so many more patches due to -mm)

protex · Post by **protex** » Tue Apr 04, 2006 10:45 pm

Aye. This wasn't meant to start an argument of any kind, just let me go on record as saying that.

As for the benchmarks, I'm really pressed for time and still a Linux newbie so I might not be the best person to try it out, but anyone else who wishes too, please post the results here.

Thanks for the replies guys.

Jedi Master · Post by **Jedi Master** » Wed Apr 05, 2006 7:15 am

Not a real benchmark - but shifting from nitro staircase&cfq to lightno (I need suspend2 ) nickshed&anticipatory gave me a nice speedup for mini-SAP application server.

Unfortunately SAP hangs with beyond - as well as with any other patchset using starcase 13 and later. But looks like nickshed is better for this application.

Regarding the benchmarks etc - personally I think this is stupid. The performance is limited with the hardware ability - and the actual responce time of the machine depends on the way the programs interact with kernel - so nickshed will perform better for one task, but can be way worse on the other. So just choose better kernel config for the software you need - not for the bencmarks to run faster.

Tiger683 · Post by **Tiger683** » Wed Apr 05, 2006 9:26 am

Jedi Master wrote:Not a real benchmark - but shifting from nitro staircase&cfq to lightno (I need suspend2 ) nickshed&anticipatory gave me a nice speedup for mini-SAP application server.

Unfortunately SAP hangs with beyond - as well as with any other patchset using starcase 13 and later. But looks like nickshed is better for this application.

Regarding the benchmarks etc - personally I think this is stupid. The performance is limited with the hardware ability - and the actual responce time of the machine depends on the way the programs interact with kernel - so nickshed will perform better for one task, but can be way worse on the other. So just choose better kernel config for the software you need - not for the bencmarks to run faster.

Obviously, you should know that for server applications the cks patchset is (muuuuuuuuch) more suitable than the
desktop/interactivity oriented ck patchset. beyond-dev has (since -git4) a Kconfig option to simply activate the tweaks from cks.
This is going into beyond2 as well (most likely). You might want to give it a try.
Nevertheless, the point is choosing the right scheduler AND it's right flavour for the corresponding applications.

T

Jedi Master · Post by **Jedi Master** » Wed Apr 05, 2006 9:31 am

Yes, I know about cks - the trick is that with mini-SAP you are ending up in a one box system - so you have server and client running on the same machine - of course you can have GUI running on other box and be happy if you have several PC's at home

. So you have to run a pretty heavy server application and at the same time pretty heavy java GUI - getting to a pure server patchset will turn your GUI life into a nightmare

But in case of serious server app you are completely right - cks is the good option, and it's very nice to have it on beyound patchset.

_loki_ · Post by **_loki_** » Fri Apr 07, 2006 9:31 am

well I'm curios too.. Can someone explain or provide a link to explain what the major difference between:
1.) CPU shedulers (Staircase vs. Nickshed)
2.) I/O Shedulers (CFQ vs. Anticipatory)
is after all. Maybe that would put people like me in a better position to decide what is the best individual choice for them. BTW I'm having installed both: beyond and no and both running quite smooth, thanks for the work, all of you!

nbvcxz · Post by **nbvcxz** » Fri Apr 07, 2006 10:39 am

_loki_ wrote:well I'm curios too.. Can someone explain or provide a link to explain what the major difference BTW I'm having installed both: beyond and no and both running quite smooth, thanks for the work, all of you!

The same as me. Few days ago I installed no-sources at my gentoo range - P3 600 and 128 ram (old computer - easy to see differences in performance). So I can't see any differences in responsiveness and overall performanece. The one thing I can see is that OpenOffice is starting few sec. faster at no-sources.
But it would good idea to have one topic about gaining performane with different optimalisations (not only in kernel). I am not talking about -O3 --super-ricer-enable

but about ideas for making linux smoother and faster as desktop.

cheater1034 · Post by **cheater1034** » Fri Apr 07, 2006 8:29 pm

IO Scheduler?:
I would only use CFQ for ext3, never on reiser4 of course, as it causes latencies to go through the roof.
I am not so much a fan of anticapatory anymore, it may be better on older systems, Deadline is my favorite, it is lighter than anticapatory, and works better in most cases.

CPU Scheduler?:
I ran a benchmark of staircase and nicksched, I will share the results
The kernel is 2.6.16-no1, I ported staircase v15 to it (from the newer plugsched)
System:
2.6.16-no1 (Upgraded plugsched for staircase v15)
AMD Athlon64 2800+ (1.8ghz Default Clock)
1 gb ddr 333 ram (well, running at ddr333, 1 is a ddr400 stick)
Gigabye k8ns pro
Maxtor sata150 200gb hdd (Reiser4 as root FS)
Results: (Nicksched-Staircase.v15)

Code: Select all

No-Sources - CPU Scheduler Interactivity Benchmark (interbench 0.3)
===================================================================
----
Kernel: 2.6.16-no1
Scheduler: Nicksched
----
--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.002 +/- 0.00241    0.012             100            100
Video     0.002 +/- 0.00236    0.003             100            100
X         0.003 +/- 0.0288     0.703             100            100
Burn      0.002 +/- 0.00251    0.009             100            100
Write     0.008 +/- 0.0103     0.067             100            100
Read       0.03 +/- 0.228        3.4             100            100
Compile    0.87 +/- 1.82        41.4             100            100
Memload   0.011 +/- 0.0543      1.04             100            100

--- Benchmarking simulated cpu of Video in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.002 +/- 0.00275    0.012             100            100
X           2.6 +/- 10.6        81.3            92.7           90.2
Burn      0.686 +/- 3.38        16.7             100           95.9
Write       6.9 +/- 10.7        16.7              99           58.6
Read      0.054 +/- 0.319       4.16             100            100
Compile   0.074 +/- 1.034       31.2             100           98.3
Memload   0.063 +/- 0.454       10.2            100            100


--- Benchmarking simulated cpu of X in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.023 +/- 0.208          2            99.3           98.3
Video      14.3 +/- 27            68            35.5           26.3
Burn      0.007 +/- 0.087          1             100           99.8
Write      6.78 +/- 37.6         390            40.6           38.2
Read      0.013 +/- 0.141          2             100             99
Compile   0.524 +/- 3.11          42            92.1           91.6
Memload   0.054 +/- 0.309          3            96.5           96.2
--- Benchmarking simulated cpu of Gaming in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
None          0 +/- 0.00364    0.063             100
Video      65.1 +/- 65.9        73.2            60.6
X          83.4 +/- 126          385            54.5
Burn        369 +/- 751         1600            21.3
Write      95.3 +/- 191          521            51.2
Read       9.36 +/- 9.57        13.4            91.4
Compile     475 +/- 837         2084            17.4
Memload    15.5 +/- 47           566            86.6

----
Kernel: 2.6.16-no1
Scheduler: Staircase v15
----
--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.002 +/- 0.00256    0.005             100            100
Video     0.002 +/- 0.0027     0.007             100            100
X         0.235 +/- 1.08          10             100            100
Burn      0.032 +/- 0.532         11             100            100
Write     0.102 +/- 1.25        23.4             100            100
Read      0.016 +/- 0.126       3.04             100            100
Compile   0.076 +/- 1.55        37.7             100            100
Memload   0.011 +/- 0.0525     0.956             100            100

--- Benchmarking simulated cpu of Video in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.002 +/- 0.00276    0.032             100            100
X         0.219 +/- 1.09        16.7             100           99.9
Burn      0.002 +/- 0.00258    0.005             100            100
Write      6.01 +/- 10          16.7            99.5             64
Read      0.028 +/- 0.221       3.67             100            100
Compile   0.054 +/- 0.934       27.7             100           99.9
Memload   0.041 +/- 0.366          9             100            100
--- Benchmarking simulated cpu of X in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None       0.55 +/- 4.24          36            97.4             96
Video      17.1 +/- 31.1         124            33.9             24
Burn      0.006 +/- 0.0817         1             100           99.3
Write      2.07 +/- 6.23          26            78.5           70.8
Read      0.013 +/- 0.141          2             100             99
Compile   0.419 +/- 2.81          35            94.4           92.9
Memload   0.044 +/- 0.296          2            98.1             97

--- Benchmarking simulated cpu of Gaming in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
None      0.077 +/- 0.698       10.1            99.9
Video      66.2 +/- 66.5        73.2            60.2
X           101 +/- 203         1221            49.9
Burn        385 +/- 466          560            20.6
Write       130 +/- 177          493            43.5
Read       3.09 +/- 4.89        17.7              97
Compile     567 +/- 746         2390              15
Memload    8.62 +/- 29.2         202            92.1

It is somewhat a nail biting contest, nicksched seems to lose by a lot in a few categories, and staircase loses in others. Overall, the results seem to show on my system, nicksched is my preference.
(Nick Piggins works on ingosched now, which is originally by ingo molnar, you can check it out and get some of his work in there)

Ma3oxuct · Post by **Ma3oxuct** » Sat Apr 08, 2006 3:47 am

These benchmarks might be a bit scewd...
Staircase produced a disk i/o problem that has recently been fixed in ck3. This fix really drives the benchmark in favor of CK.

Do not take my word for it, as I might be getting things confused here. I do not have time right now to look into the possible issue.

bollucks · Post by **bollucks** » Sat Apr 08, 2006 6:11 am

Ma3oxuct wrote:These benchmarks might be a bit scewd...
Staircase produced a disk i/o problem that has recently been fixed in ck3. This fix really drives the benchmark in favor of CK.

Do not take my word for it, as I might be getting things confused here. I do not have time right now to look into the possible issue.

Staircase v15 included here has the disk i/o fix so these are current benchmarks.

Thetargos · Post by **Thetargos** » Wed Jul 19, 2006 10:49 pm

Have you tested again with 2.6.17? What application did you use to benchmark, and what methodology?

darklegion · Post by **darklegion** » Thu Jul 20, 2006 1:33 am

Although the later versions of staircase are much better than the older versions, it still has its problems; audio being the most noticable.For instance I often run games in a windows guest under vmware.While playing it works fine without too much audio skipping under staircase (although possibly more than ingosched or nicksched) when I switch out of vmware temporarily to do some browsing with opera, changing tabs results in heavy audio skipping.This does not happen with ingo or nick schedulers.This is not a big issue, but it does prove that staircase still has some weaknesses.

EDIT : BTW I should mention I have played around with changing priorities with schedtool, but this did not help

Thetargos · Post by **Thetargos** » Thu Jul 20, 2006 1:52 pm

I ran into a problem with intebench in that if running in non-realtime mode the memory load test would never finish (and I left the computer "do its thing" for almost 9 hours!). Oddly enough the numbers were strange, if running in RT mode the latencies would be higher than in normal mode, especially the write test.