View previous topic :: View next topic |
Author |
Message |
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 5:50 am Post subject: High timer interrups (INT 0) with 3.4.9 |
|
|
I have three machines, two of which I recently upgraded to 3.4.9 and just noticed that they have extremly high timer interrupt count (INT 0) - one reached 10 millions
form Sept 15th, the other - 500 millions from August 27th reboot. At the same time, the only machine that stayed at 2.6.39 kernel, shows just 6000 timer interrupts
since its boot on August 1st.
What could be the reason ? I don't think I played with kernel config options that much (or differently). All machines are quite different - high interrupt count one are
Core 2 Dupo laptop and i7 desktop, Intel and nvidia video correspodingly, the one with low count is core2 duo desktop with nvidia video. So it seems it is a kernel version which matters |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Tue Sep 18, 2012 7:32 am Post subject: |
|
|
Did you have tickless kernel enabled? Is it incrementing like a hundred every second or whatever your HZ is?
This was fairly common and normal in old Linux kernels that don't support tickless...after enabling tickless it's much lower... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 2:36 pm Post subject: |
|
|
eccerr0r wrote: | Did you have tickless kernel enabled? Is it incrementing like a hundred every second or whatever your HZ is?
This was fairly common and normal in old Linux kernels that don't support tickless...after enabling tickless it's much lower... |
I have tickless system, I double checked ( CONFIG_NO_HZ=y ) . Although 3.4.9 kernels have an additional option, CONFIG_RCU_FAST_NO_HZ and this one is not set. 2.6.39 does not have it.
I timed the interrupts - I get approximately 500 interrupts per second, and system timer is at 1000 HZ, so it increments at twice a slower rate. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Tue Sep 18, 2012 2:55 pm Post subject: |
|
|
Is it like this on fresh boot? Single user mode?
Do you have any special hardware? Can the drivers be removed one at a time to see if any of the drivers require frequent interrupts by the clock tick timer?
Do you have HPET enabled on the newer machines? APIC?
Hmm... thought I had a 3.4.9 machine, and doesn't seem to be exhibiting interrupt storm in hw INT0... But I get a lot of local interrupts...
Code: | mikuru $ uname -a
Linux mikuru 3.4.9-gentoo #2 SMP Sun Aug 26 09:37:03 MDT 2012 x86_64 Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz GenuineIntel GNU/Linux
mikuru $ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 127 0 0 0 IO-APIC-edge timer
1: 292 9536 146 311 IO-APIC-edge i8042
8: 2 11 0 0 IO-APIC-edge rtc0
9: 250 205 5 85 IO-APIC-fasteoi acpi
12: 7240 260815 3375 7179 IO-APIC-edge i8042
16: 84 132 30 38 IO-APIC-fasteoi ehci_hcd:usb1
23: 0 23 5 4 IO-APIC-fasteoi ehci_hcd:usb2
40: 5886 9660 3690 2082 PCI-MSI-edge i915
41: 19673 16300 4217 5245 PCI-MSI-edge ahci
42: 12821 30427 9406 4772 PCI-MSI-edge xhci_hcd
43: 6457 24409 3702 3569 PCI-MSI-edge snd_hda_intel
44: 0 0 0 0 PCI-MSI-edge eth0
45: 70034 267897 31547 40622 PCI-MSI-edge iwlwifi
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 688301 412667 713302 374754 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RTR: 3 0 0 0 APIC ICR read retries
RES: 389680 266397 32705 15704 Rescheduling interrupts
CAL: 67 82 73 77 Function call interrupts
TLB: 11227 13067 8305 12849 TLB shootdowns
TRM: 8 8 8 8 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 19 19 19 19 Machine check polls
ERR: 0
MIS: 0
|
_________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 2:58 pm Post subject: |
|
|
Good questions, I'll try to invesitage. It is a run-of-the-mill desktop, but has e-sata external drive attached. Last time it was rebooted on Sept 5th.
I'll play today with my laptop which shows the same, and for which I still have 2.6.39 kernel to boot in to compare. |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 4:17 pm Post subject: |
|
|
And, havn't yet done the investigation, my interrupt table is
Code: |
# uname -a
Linux wheeler 3.4.9-gentoo #1 SMP Mon Aug 27 14:13:10 MDT 2012 x86_64 Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz GenuineIntel GNU/Linux
#cat /proc/interrupts
0: 116 0 0 541481683 0 0 0 0 IO-APIC-edge timer
1: 0 0 0 159774 0 0 0 0 IO-APIC-edge i8042
8: 0 0 0 64 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 9190560 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
17: 0 0 0 0 10746994 0 0 0 IO-APIC-fasteoi ahci, snd_hda_intel
18: 0 0 0 0 0 102422837 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8, firewire_ohci
19: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
44: 12579660 0 0 0 0 0 0 0 PCI-MSI-edge ahci
45: 0 37480974 0 0 0 0 0 0 PCI-MSI-edge eth0
47: 0 0 407490 0 0 0 0 0 PCI-MSI-edge snd_hda_intel
48: 0 0 0 118904701 0 0 0 0 PCI-MSI-edge nvidia
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 185727185 200928325 181580118 82962887 52201164 48264262 66709717 80949762 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RTR: 6 0 0 0 0 0 0 0 APIC ICR read retries
RES: 142684187 40531377 4161936 1538613 345423 289461 147759 164673 Rescheduling interrupts
CAL: 6568138 5452265 6344138 3496687 7217879 7221072 7226884 7222810 Function call interrupts
TLB: 1779471 1784771 1826604 1847394 1851097 1850794 1902552 970374 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 6287 6287 6287 6287 6287 6287 6287 6287 Machine check polls
ERR: 0
MIS: 0
|
And your's is looking as I would expect it to look |
|
Back to top |
|
|
aCOSwt Bodhisattva
Joined: 19 Oct 2007 Posts: 2537 Location: Hilbert space
|
Posted: Tue Sep 18, 2012 4:22 pm Post subject: |
|
|
Being said that my system gets a far higher interrupt rate than yours (1.500/s), I can't see any significant difference between my 2.6.38 and 3.4.9
Which CONFIG_HZ_xyzt did you set ? _________________
|
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 4:45 pm Post subject: |
|
|
aCOSwt wrote: | Being said that my system gets a far higher interrupt rate than yours (1.500/s), I can't see any significant difference between my 2.6.38 and 3.4.9
Which CONFIG_HZ_xyzt did you set ? |
Mine is CONFIG_HZ_1000 |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Tue Sep 18, 2012 4:48 pm Post subject: |
|
|
I have to mention that when I first posted it, it's been up only an hour or so. As of right now, it's up 3 hours 17 minutes and hasn't changed from the 127 count... Everything else has gone up though.
Weird.
INT0 should be the main interval timer...
CONFIG_NO_HZ=y
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_HPET=y
CONFIG_X86_LOCAL_APIC=y
update--
I just looked on my core2 quad machine and it's acquiring about 40 interrupts per core per second...Not quite the same issue but probably similar. My core2 duo machine is holding fairly steady on INT0 however.
The core2 quad machine has closed source software though (FGLRX).
It does have HZ=1000, not have CONFIG_HPET set, but it's running Linux-3.3.8-gentoo ...
update 2--
You guys made me boot my i7-2700k ... It's currently running 3.2.21-gentoo.
It does not appear to be getting INT0 bloat like the core2quad, but HPET=YES.
So for me it looks like HPET made the difference... Granted I should try 3.4.9 though... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 7:36 pm Post subject: |
|
|
eccerr0r wrote: |
update--
I just looked on my core2 quad machine and it's acquiring about 40 interrupts per core per second...Not quite the same issue but probably similar. My core2 duo machine is holding fairly steady on INT0 however.
The core2 quad machine has closed source software though (FGLRX).
It does have HZ=1000, not have CONFIG_HPET set, but it's running Linux-3.3.8-gentoo ...
update 2--
You guys made me boot my i7-2700k ... It's currently running 3.2.21-gentoo.
It does not appear to be getting INT0 bloat like the core2quad, but HPET=YES.
So for me it looks like HPET made the difference... Granted I should try 3.4.9 though... |
One both my offending machines I have HPET=YES, but dmesg says
Code: |
ACPI: HPET 00000000afee6c00 00038 (v01 GBT GBTUACPI 42302E31 GBTU 00000098)
ACPI: HPET id: 0x8086a201 base: 0xfed00000
HPET: 4 timers in total, 0 timers will be used for per-cpu timer
|
(other offending machine has 3 timers is total, but again 0 timers will be used for per-cpu timer)
Don't know what it means and I have to reboot to see what HPET is doing on the one machine that does not show high INT0
I remember very clearly how INT0 interrupts disappeared on all my machines when tickless kernels first appear few years ago.
I am also somewhat suprised about high counts of INT18 - on which some USB controllers sit. It is also at the level of 100 per second.
And I have just USB mouse and camera (almost always off), and SD controller (not used as well). |
|
Back to top |
|
|
aCOSwt Bodhisattva
Joined: 19 Oct 2007 Posts: 2537 Location: Hilbert space
|
Posted: Tue Sep 18, 2012 7:44 pm Post subject: |
|
|
Before understanding where the difference could come from, you can test the actual capabilities of your different systems in terms of interrupt rates.
(no need to single user mode...)
Code: |
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#define USECREQ 250
#define LOOPS 1000
void event_handler (int signum){
static unsigned long cnt = 0;
static struct timeval tsFirst;
if (cnt == 0){
gettimeofday (&tsFirst, 0);}
cnt ++;
if (cnt >= LOOPS){
struct timeval tsNow;
struct timeval diff;
setitimer (ITIMER_REAL, NULL, NULL);
gettimeofday (&tsNow, 0);
timersub(&tsNow, &tsFirst, &diff);
unsigned long long udiff = (diff.tv_sec * 1000000) + diff.tv_usec;
double delta = (double)(udiff/cnt)/1000000;
int hz = (unsigned)(1.0/delta);
printf ("kernel timer interrupt frequency is approx. %d Hz", hz);
if (hz >= (int) (1.0/((double)(USECREQ)/1000000))){
printf (" or higher");}
printf ("\n");
exit (0);}}
int main (int argc, char **argv){
struct sigaction sa;
struct itimerval timer;
memset (&sa, 0, sizeof (sa));
sa.sa_handler = &event_handler;
sigaction (SIGALRM, &sa, NULL);
timer.it_value.tv_sec = 0;
timer.it_value.tv_usec = USECREQ;
timer.it_interval.tv_sec = 0;
timer.it_interval.tv_usec = USECREQ;
setitimer (ITIMER_REAL, &timer, NULL);
while (1);} |
(credits advenage)
Copy, paste, save, and gcc this.
On my current core 2 duo running ck-sources-3.4.9 with CONFIG_HZ=1000 / CONFIG_NOHZ=y / CONFGI_HPET* =y, I get :
Code: | acoswt@PrimaPratica ~ $ ./a.out
kernel timer interrupt frequency is approx. 4016 Hz or higher |
_________________
|
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Tue Sep 18, 2012 7:52 pm Post subject: |
|
|
aCOSwt wrote: |
On my current core 2 duo running ck-sources-3.4.9 with CONFIG_HZ=1000 / CONFIG_NOHZ=y / CONFGI_HPET* =y, I get :
Code: | acoswt@PrimaPratica ~ $ ./a.out
kernel timer interrupt frequency is approx. 4016 Hz or higher |
|
On that one 'offending' i7 machine I am at right now (gentoo-sources-3.4.9 wth CONFIG_HZ=1000 / CONFIG_NOHZ=y / CONFGI_HPET* =y)
I get exactly the same
kernel timer interrupt frequency is approx. 4016 Hz or higher
Not sure about #define USECREQ 250, could it be we should change it to 1000 ? with this redifinition I get
kernel timer interrupt frequency is approx. 1002 Hz or higher which makes more sense |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Tue Sep 18, 2012 8:43 pm Post subject: |
|
|
This code appears to be measuring the precision of the timer, but because of tickless kernel, the timer may not be running at all times. There's something weird going on here... (Mine is also saying about 4KHz for that code, despite machines that are not triggering thousands of INT0's.)
I'm out of ideas now, could you post your .config's for correct and incorrect behavior? Maybe someone could scrutinize them for differences... http://users.vanade.com/~eccerr0r/mikuru.config has my .config for my i5 (now up 7 hours and still at 127 INT0s) that seems to keep counts low if interested...
(and there's a specific reason for interest... though performance might not really matter, my i5 is a laptop and reducing the number of interrupts equates to better battery life...) _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Wed Sep 19, 2012 2:06 am Post subject: |
|
|
eccerr0r wrote: |
I'm out of ideas now, could you post your .config's for correct and incorrect behavior? Maybe someone could scrutinize them for differences... http://users.vanade.com/~eccerr0r/mikuru.config has my .config for my i5 (now up 7 hours and still at 127 INT0s) that seems to keep counts low if interested...
(and there's a specific reason for interest... though performance might not really matter, my i5 is a laptop and reducing the number of interrupts equates to better battery life...) |
This is my main concern as well, since one of two machines with high INT0 is a laptop. I however just run powertop on it, and there is no trace that timer interrupts register in CPU wakeups. On a clean boot, with just kdm running, no network, no wireless, powertop shows 25 wakeups per second, with leading cause being swapper/0,
then kdm, then 'rescheduling interrupts'. But nothing like hundreds per second timer wakeups. CPU is nicely in C3 state >99% of the time.
I wonder, how to reconcile powertop and /proc/interrupts reading.
BTW, I booted into older 2.6.38-2.6.39 kernels on both machines, and they show as high timer count. Still a puzzle why my last machine shows none of those. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Wed Sep 19, 2012 4:24 am Post subject: |
|
|
I have a feeling some of those /proc/interrupts are double counted in that file, maybe powertop knows now to deal with the doublecounting.
And oddly enough I can simply not get powertop to get below 10 interrupts/second for some reason with Gentoo (it will print as green). Even with everything quiescent best I get was like 20 or so (brown). When in the 30s it will print in red.
For sure need to disable wireless, USB devices, ethernet, etc. to get it down.
But I still don't see how these affect INT0... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Wed Sep 19, 2012 4:56 am Post subject: |
|
|
eccerr0r wrote: | I have a feeling some of those /proc/interrupts are double counted in that file, maybe powertop knows now to deal with the doublecounting.
And oddly enough I can simply not get powertop to get below 10 interrupts/second for some reason with Gentoo (it will print as green). Even with everything quiescent best I get was like 20 or so (brown). When in the 30s it will print in red.
For sure need to disable wireless, USB devices, ethernet, etc. to get it down.
But I still don't see how these affect INT0... |
based on the description I googled, powertop gets its info from /proc/timer_stats. There I do not see reference to INT0 timer as well. Say, my current is
Code: |
cat /proc/timer_stats
Timer Stats Version: v0.2
Sample period: 0.507 s
8, 0 swapper/1 tick_nohz_idle_exit (tick_sched_timer)
1D, 8 kworker/1:0 do_dbs_timer (delayed_work_timer_fn)
3, 0 swapper/0 tick_nohz_stop_sched_tick.clone.7 (tick_sched_timer)
3D, 1046 kworker/0:2 do_dbs_timer (delayed_work_timer_fn)
1D, 1046 kworker/0:2 process_one_work (delayed_work_timer_fn)
3, 0 swapper/0 run_timer_softirq (cursor_timer_handler)
1, 1832 ifplugd schedule_hrtimeout_range_clock (hrtimer_wakeup)
20 total events
|
so it is at most 20 events per half a second |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Wed Sep 19, 2012 5:02 am Post subject: |
|
|
eccerr0r wrote: | I have a feeling some of those /proc/interrupts are double counted in that file, maybe powertop knows now to deal with the doublecounting.
And oddly enough I can simply not get powertop to get below 10 interrupts/second for some reason with Gentoo (it will print as green). Even with everything quiescent best I get was like 20 or so (brown). When in the 30s it will print in red.
For sure need to disable wireless, USB devices, ethernet, etc. to get it down.
But I still don't see how these affect INT0... |
I don't think I was ever able to go below 10 as well. Currently , if I am in kde, but with no wireless/network, firefox not launched, and I don't touch keyboard/mouse, I am getting 10-12 wakeups/per sec, with some due to kwin and some due to LOC (Local timer interrupts). I remember when I played with
powertop a lot couple of years ago, it were these LOC's that was giving me minumum that I could not beat.
However and remarkably, if a switch from KDE to console (ALT-CNTRL-F1) I get twice more, 20 wakeups/s - frome somewhere swapper/0 appears that maintains this extra 10. Go figure why it is not present when I am inside KDE |
|
Back to top |
|
|
aCOSwt Bodhisattva
Joined: 19 Oct 2007 Posts: 2537 Location: Hilbert space
|
Posted: Wed Sep 19, 2012 9:55 am Post subject: |
|
|
dmpogo wrote: |
Not sure about #define USECREQ 250, could it be we should change it to 1000 ? with this redifinition I get
kernel timer interrupt frequency is approx. 1002 Hz or higher which makes more sense |
You put the value you want. I mean irrespective of the value of CONFIG_HZ you kernel was built with.
The purpose of the code listed above is to determine the capabilities of the timer, not to determine the frequency at which the kernel requests it to run.
So, with USECREQ (microseconds request) you define the value of the interval of time (timer.it_interval.tv_usec = USECREQ;) between two timer events.
In the example given : 250µs => corresponding to a 4000Khz frequency.
And the answer when you ran it was : Yes it can... and probably even higher.
If, on my system I want to check with USECREQ=1, the answer is yes again it can.
What I had suggested you was to run this on the system under which you see a very low number of interrupts, just in order to check if this system is actually capable of much more.
In a near to tickless system the value given as CONFIG_HZ is not the value your system will necessarily use. It is a sort of fallback.
In a totally tickless system, each time an event occurs, the kernel simply programs the timer to trigger an interrupt when the next earliest event is due irrespective of any potential cadenza.
The problem for going completely with this approach is that the kernel still needs to keep a certain track of relative time. So there still must be some period ticks somewhere.
On my system (two cores) with CONFIG_HZ=1000, I can sometimes see more than 3500 int0 per second. That is to say above the CONFIG_HZ value.
It averages around 1500, but I am running JACK. As soon as I kill JACK, the average looses the 1000hz rate jack was asking for.
In conclusion, provided you ensured that your low-int0-rate system is capable of a greater one, then the answer is that the difference must comes from what you are actually running on these systems. Is one most of the times iddle when the other one permanently running some audio server, multimedia player...?
Anyway, provided you ensured your machine is capable of more, then, contrarily to what you suggested in your first post, the discrepency is not kernel dependent.
EDIT : As a consequence of all this, selecting CONFIG_NO_HZ on your system heavily interrupted will not help you saving whatever power.
But it does save significant energy on the other one. _________________
|
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Wed Sep 19, 2012 2:10 pm Post subject: |
|
|
aCOSwt wrote: | dmpogo wrote: |
Not sure about #define USECREQ 250, could it be we should change it to 1000 ? with this redifinition I get
kernel timer interrupt frequency is approx. 1002 Hz or higher which makes more sense |
You put the value you want. I mean irrespective of the value of CONFIG_HZ you kernel was built with.
The purpose of the code listed above is to determine the capabilities of the timer, not to determine the frequency at which the kernel requests it to run.
So, with USECREQ (microseconds request) you define the value of the interval of time (timer.it_interval.tv_usec = USECREQ;) between two timer events.
In the example given : 250µs => corresponding to a 4000Khz frequency.
And the answer when you ran it was : Yes it can... and probably even higher.
If, on my system I want to check with USECREQ=1, the answer is yes again it can.
What I had suggested you was to run this on the system under which you see a very low number of interrupts, just in order to check if this system is actually capable of much more.
In a near to tickless system the value given as CONFIG_HZ is not the value your system will necessarily use. It is a sort of fallback.
In a totally tickless system, each time an event occurs, the kernel simply programs the timer to trigger an interrupt when the next earliest event is due irrespective of any potential cadenza.
The problem for going completely with this approach is that the kernel still needs to keep a certain track of relative time. So there still must be some period ticks somewhere.
On my system (two cores) with CONFIG_HZ=1000, I can sometimes see more than 3500 int0 per second. That is to say above the CONFIG_HZ value.
It averages around 1500, but I am running JACK. As soon as I kill JACK, the average looses the 1000hz rate jack was asking for.
In conclusion, provided you ensured that your low-int0-rate system is capable of a greater one, then the answer is that the difference must comes from what you are actually running on these systems. Is one most of the times iddle when the other one permanently running some audio server, multimedia player...?
Anyway, provided you ensured your machine is capable of more, then, contrarily to what you suggested in your first post, the discrepency is not kernel dependent.
EDIT : As a consequence of all this, selecting CONFIG_NO_HZ on your system heavily interrupted will not help you saving whatever power.
But it does save significant energy on the other one. |
Right, tried it on the machine that accumulated just 250 INT0 interrupts since it was rebooted 10 hours ago and the code gives same 4016 or higher.
It runs netatalk, backuppc (having performed backups within this 10 hours) and idle KDE session as we speak
As of what I run on machines with high (and persistent) INT0.
One is the laptop, fresh reboot, just logged in KDE, run wireless and firefox, uptime 28 minutes - 200000 interrupts, 120 interrupts/sec. It was the same in last try when I did not log into KDE or switched on wireless, just sat in console. It reached 320000 as I was typing this. These interrupts do not however register in powertop as wakeups.
The other is office desktop - run kde, some nfs disks, backuppc, numerical code from time to time, external sata drive for backup. It sits at 170 INT0 interrupts per second solid, since reboot yesterday 18 hours ago.
Also you are right that the difference is not kernel dependent, at least kernel version dependent. On both machines with high interrupts I booted in the old kernels, 2.6.39 and 2.6.38 and they showed the same high INT0. So this hypothesis can be put to rest. Kernel configs are, however different on all three machines, so there maybe a difference here, but I could not visualize the pattern. One thing common between high INT0 machines is that they use deadline scheduler and scheduler is built in, while low INT0 uses CFQ. Mere runtime switching to CFQ did not help my laptop, however maybe just compiling deadline into kernel causes some timer related code to run ?
Hardware also does not point to a pattern
Low INT0 - core 2 duo, nvidia, snd_hda_intel, no wifi, realtek ethernet
High INT0 - a) core 2 duo, intel graphics, wifi, snd_hda_intel, intel ethernet (off) laptop
b) 4 core i7, nvidia, snd_hda_intel, realtek ethernet, desktop
PS of course it can be coincidence, that there is some process, like wifi, which causes timer to run at 120 INT0/s on laptop, and something else on desktop.
I just tried to switch to CFQ with no deadline compiled in, and INT0 seems slowed to 35/s, but after longer run with wifi on it is back to 122/s |
|
Back to top |
|
|
aCOSwt Bodhisattva
Joined: 19 Oct 2007 Posts: 2537 Location: Hilbert space
|
Posted: Wed Sep 19, 2012 3:26 pm Post subject: |
|
|
dmpogo wrote: | One thing common between high INT0 machines is that they use deadline scheduler and scheduler is built in, while low INT0 uses CFQ. |
I never used CFQ, moreover I use a ck-patched kernel which does its best in order to reduce the IO throughput, so I cannot really tell about this.
However, I used to have noop and deadline compiled into my kernel, switching from one to the other for testing purposes, then I realized deadline did not help so I no longer build the deadline scheduler.
I never watched any significant difference in terms of int0 rate between deadline and noop with deadline built, nor did I watch any significant difference since I no longer build deadline.
dmpogo wrote: | Hardware also does not point to a pattern
Low INT0 - core 2 duo, nvidia, snd_hda_intel, no wifi, realtek ethernet
High INT0 - a) core 2 duo, intel graphics, wifi, snd_hda_intel, intel ethernet (off) laptop
b) 4 core i7, nvidia, snd_hda_intel, realtek ethernet, desktop |
I do not think the answer is to be searched around these.
Perhaps in the daemons running, almost certainly around X, kde & related qt apps.
Akgregator, Amarok... have had long histories of triggering high int0 rates when iddle.
You just cannot imagine the number of threads doing nothing else but polling as soon as these apps are launched.
And polling requires... timer interrupts.
And then depending on X, apps, qt, kde versions, polling rates can be increased, decreased... _________________
|
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Wed Sep 19, 2012 4:52 pm Post subject: |
|
|
I run Gnome on pretty much all machines, network manager and all. Even blueman and the polling gweather, gnome-power-manager, etc.
The thing is, I thought that INT0 may only be used during boot or something. I reboot my i5 and it still came back up as 127 somehow.
My wifi does not contribute to INT0 counts (of course), just its corresponding MSI.
I wonder what the interrupt numbers would look like if you compiled the same kernel as mine?
The hardware on my laptop: Intel Graphics (HD4000), realtek gbit ethernet, intel wifi 2230N, Z77 chipset; it's a pretty basic laptop (HP Envy4t). _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Wed Sep 19, 2012 5:11 pm Post subject: |
|
|
aCOSwt wrote: |
Perhaps in the daemons running, almost certainly around X, kde & related qt apps.
Akgregator, Amarok... have had long histories of triggering high int0 rates when iddle.
You just cannot imagine the number of threads doing nothing else but polling as soon as these apps are launched.
And polling requires... timer interrupts.
And then depending on X, apps, qt, kde versions, polling rates can be increased, decreased... |
What actually is practically identical between three machines is KDE setup. And it is -semantic-desktop on all three - so no nepomunks, amarok's
or anything like that even installed. It is as barebone KDE setup as I could get. Well it has weather applet - on all three machines, it has UPS and its applet on two - one with low INT0, one with high (desktops), X ? one high INT machine uses nvidia, the other - intel, low int machine is nvidia. External eSata drives ? both on low INT and high INT desktops, none on high INT laptop.
So go figure. One thing in common that both high INT machines have SSD drives (that's why I suspected deadline), but I'd suprised it has anything to do.
Finding what may poll is an interesting point - that's I believe why I have high interrupt counts on one USB device, it is probably usbdisks polling it all the time |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3266 Location: Canada
|
Posted: Wed Sep 19, 2012 5:13 pm Post subject: |
|
|
eccerr0r wrote: | I run Gnome on pretty much all machines, network manager and all. Even blueman and the polling gweather, gnome-power-manager, etc.
The thing is, I thought that INT0 may only be used during boot or something. I reboot my i5 and it still came back up as 127 somehow.
My wifi does not contribute to INT0 counts (of course), just its corresponding MSI.
I wonder what the interrupt numbers would look like if you compiled the same kernel as mine?
The hardware on my laptop: Intel Graphics (HD4000), realtek gbit ethernet, intel wifi 2230N, Z77 chipset; it's a pretty basic laptop (HP Envy4t). |
What I'll try to do (need to find time) is copy a kernel config from my high INT desktop to a low INT one. They have similar hardware so it can be a close comparison. |
|
Back to top |
|
|
aCOSwt Bodhisattva
Joined: 19 Oct 2007 Posts: 2537 Location: Hilbert space
|
Posted: Wed Sep 19, 2012 6:59 pm Post subject: |
|
|
eccerr0r wrote: | So for me it looks like HPET made the difference |
I thought HPET was set for both configs.
But I realize that even then, there might well be some difference because of the BIOS :
HPET Legacy Replacement Route option (hpet_lrr) wrote: |
+
+HPET is capable of replacing the IRQ0 (connected INT0 PIN) routing for
+timer interrupt. The capability register (at offset 0 of HPET
+base address) has a bit specifying if HPET chip is capbale of doing
+this. OS can read the bit either from HW or ACPI table. (HPET ACPI
+description table -> Event Timer block -> bit 15, page 30 of HPET
+spec). Ideally (I think so!) BIOS should set the ACPI table than letting
+the OS read H/W, which gives the BIOS a way to configure either legacy
+or Legacy replacement modes.
+
+Typically the motherboard has BIOS configured / hardwired IRQ0 to INT0
+(pin of APIC) connection. Linux assumes IRQ0 connected to INT0 unless it is
+supplied using an override parameter in the MPTable. Some NVidia chipsets /
+BIOS initialization code had configured to override IRQ0 -> INT0 connection
+and later a parameter was introduce (acpi_skip_timer_override) to get IRQ0 ->
+INT0 connection right.
+
+But a number of bioses (both phoenix and AMI) are not working as
+expected. (I have an AMI BIOS which sets ACPI table bit 15 to 0 and then
+connect IRQ0 -> INT2 internally, Another bios I have sets the ACPI table bit
+15 to 0, but does not connect IRQ0 -> INT2. |
Out of interest what can you read concerning hpet in your system log ?
On mine : Code: |
HPET: 4 timers in total, 0 timers will be used for per-cpu timer
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 comparators, 64-bit 14.318180 MHz counter
Switching to clocksource hpet |
_________________
|
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9675 Location: almost Mile High in the USA
|
Posted: Wed Sep 19, 2012 7:48 pm Post subject: |
|
|
Hmm... It's probably not HPET: I just looked at one of my really ancient machines that does not have HPET and it's also got very few INT0
Code: | doujima:/var/log$ cat /proc/interrupts
CPU0
0: 45 IO-APIC-edge timer
1: 23269 IO-APIC-edge i8042
4: 3759111 IO-APIC-edge
6: 5 IO-APIC-edge floppy
7: 147257 IO-APIC-edge parport0
9: 0 IO-APIC-fasteoi acpi
12: 1026863 IO-APIC-edge i8042
14: 21261716 IO-APIC-edge pata_sis
15: 20630591 IO-APIC-edge pata_sis
16: 23030 IO-APIC-fasteoi uhci_hcd:usb4, radeon@pci:0000:01:00.0
17: 41882707 IO-APIC-fasteoi pata_pdc202xx_old, uhci_hcd:usb5
18: 106239203 IO-APIC-fasteoi pata_pdc2027x, ehci_hcd:usb1, SiS SI7012
19: 226003861 IO-APIC-fasteoi ohci_hcd:usb2, eth0
22: 0 IO-APIC-fasteoi eth1
23: 0 IO-APIC-fasteoi ohci_hcd:usb3
NMI: 0 Non-maskable interrupts
LOC: 2483032918 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
IWI: 0 IRQ work interrupts
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 22645 Machine check polls
ERR: 95
MIS: 0
|
This machine does not have GNOME installed, but as can be seen by the interrupt handlers, has a lot of hard drives attached (5 PATA.)
This is an old kernel though - 2.6.38. I need to update this soon... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|