| View previous topic :: View next topic |
| Author |
Message |
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
 Posted: Sat May 06, 2006 7:01 am Post subject: IRQ balancing : irqbalance - why is it required on AMD64? |
 |
|
In an ACPI system, acpi_irq_balance is default. But the IRQs are not balanced in my ACPI SMP system:
| Code: | root@co-amd-pc ~
$ cd /proc/irq;for i in *;do echo "$i $(cat $i/smp_affinity)";done
0 1
1 1
10 1
11 1
12 1
13 1
14 1
15 1
2 1
217 3
225 3
233 3
3 1
4 1
5 1
50 3
58 3
6 1
66 3
7 1
8 1
9 1
|
As you can see, 217, 225, 233,50,58,66 all have affinity to both processors. Now, I do this:
| Code: | emerge irqbalance
/sbin/irqbalance
root@co-amd-pc /proc/irq
$ cd /proc/irq;for i in *;do echo "$i $(cat $i/smp_affinity)";done
0 1
1 2
10 1
11 1
12 1
13 1
14 1
15 1
2 1
217 2
225 1
233 3
3 1
4 1
5 1
50 1
58 3
6 1
66 2
7 1
8 1
9 1
|
as you can see, now 217, 225 (both handling two devices) have affinity to different CPUs. 58, 233 handle only one device, so its fine to have them on both cpu's.
So, the question is: why acpi_irq_balance not working? |
|
| Back to top |
|
 |
didymos
Advocate
Joined: 10 Oct 2005
Posts: 4798
Location: California
|
| Posted: Sat May 06, 2006 7:06 am Post subject: |
|
|
I don't know how the kernel handles this. Is it possibly due to a screwy DSDT? Here are what I have with my system:
| Code: |
//prebalance
0 1
1 1
10 1
11 1
12 1
13 1
14 1
15 1
169 3
177 3
2 3
209 3
217 3
225 3
233 3
3 1
4 1
5 1
6 1
7 1
8 1
9 1
//postbalance
0 1
1 1
10 1
11 1
12 1
13 1
14 1
15 1
169 3
177 3
2 3
209 3
217 3
225 3
233 3
3 1
4 1
5 1
6 1
7 1
8 1
9 1
|
_________________
Thomas S. Howard |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Sat May 06, 2006 7:28 am Post subject: |
|
|
you have to wait for sometime (typically 10 seconds) before irqbalance adjusts those values.
does anybody have a scheme whereby I can get libata and eth0 on different IRQs. Right now I have:
| Code: | root@co-amd-pc /proc/irq
$ cat /proc/interrupts
CPU0 CPU1
0: 5163719 6231462 IO-APIC-edge timer
1: 2645 2421 IO-APIC-edge i8042
8: 0 0 IO-APIC-edge rtc
9: 0 0 IO-APIC-level acpi
12: 20602 54493 IO-APIC-edge i8042
14: 0 41 IO-APIC-edge ide0
15: 19 6495 IO-APIC-edge ide1
50: 140 145 IO-APIC-level ehci_hcd:usb1
58: 2 86 IO-APIC-level ohci_hcd:usb2
66: 329267 364233 IO-APIC-level nvidia
217: 17560 74816 IO-APIC-level libata, NVidia CK804
225: 1043836 113977 IO-APIC-level libata, eth0
233: 0 3 IO-APIC-level ohci1394
NMI: 487 445
LOC: 11393455 11393059
ERR: 0
MIS: 0
|
One, I don't like nvidia and disks to share the IRQ. Second, eth0 needs to be handled by only one CPU (for TCP segmentation) but as you can see with smp_affinity, 217 is aligned to both the CPUs. I can use irqbalance and have 217 going to one CPU but then I have performance problems because both libata and eth0 use CPU 0. ||ly, libata and NVIDIA (IRQ 225) use CPU 1, and hence hamper each other. What I want is default behaviour without irqbalance but with the option of putting eth0 on its own IRQ. Is it possible? |
|
| Back to top |
|
|
didymos
Advocate
Joined: 10 Oct 2005
Posts: 4798
Location: California
|
| Posted: Sat May 06, 2006 7:43 am Post subject: |
|
|
Ah, I see. Now my values are:
| Code: |
//prebalance
0 1
1 1
10 1
11 1
12 1
13 1
14 1
15 1
169 3
177 3
2 3
209 3
217 3
225 3
233 3
3 1
4 1
5 1
6 1
7 1
8 1
9 1
//postbalance
0 2
1 1
10 1
11 1
12 1
13 1
14 2
15 1
169 1
177 1
2 3
209 3
217 2
225 3
233 1
3 1
4 1
5 1
6 1
7 1
8 1
9 1
|
_________________
Thomas S. Howard |
|
| Back to top |
|
|
didymos
Advocate
Joined: 10 Oct 2005
Posts: 4798
Location: California
|
| Posted: Sat May 06, 2006 7:50 am Post subject: |
|
|
Does your eth0 device support MSI? Mine does, and it's completely associated with cpu0.
_________________
Thomas S. Howard |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA
|
| Posted: Sat May 06, 2006 6:20 pm Post subject: |
|
|
I found that irq_balance specific code is in arch/i386 and applies only to IA32. Is that the reason the IRQs are not balanced on AMD64? Or is that part of the code shared between i386 and AMD64? anyone?
PS: what is MSI btw? |
|
| Back to top |
|
|
didymos
Advocate
Joined: 10 Oct 2005
Posts: 4798
Location: California
|
| Posted: Mon May 08, 2006 4:32 am Post subject: |
|
|
Message Signalled Interrupts. It lets devices do interrupts by passing messages through the pci bus, instead of using the "out-of-band" pin-signalled IRQs (I'm leaving stuff out, but this is basically the idea. In fact, some newer devices do old style IRQs by emulating pin-signalling through the bus. The important thing is that they behave like they have actual separate pins for IRQ assertion). The thing with MSIs is that they are unique to each device, i.e. no sharing. Not many devices use them, or rather the drivers don't. A bunch of my devices are MSI capable, but only a couple are actually using them, one being my ethernet controller.
_________________
Thomas S. Howard |
|
| Back to top |
|
|
quantumsummers
Retired Dev
Joined: 24 Sep 2006
Posts: 103
Location: Earth
|
| Posted: Mon Jun 25, 2007 4:29 pm Post subject: looking around for an answer |
|
|
Hello,
I've been concerned about the same issue for some time now. I would really like to assign IRQs to eliminate sharing of big users like nvidia, eth*, and libata. Check this out:
| Code: |
cat /proc/interrupts
CPU0
0: 123742165 IO-APIC-edge timer
1: 50039 IO-APIC-edge i8042
6: 3 IO-APIC-edge floppy
8: 0 IO-APIC-edge rtc
9: 0 IO-APIC-fasteoi acpi
12: 1803385 IO-APIC-edge i8042
15: 1119319 IO-APIC-edge ide1
17: 248525 IO-APIC-fasteoi libata, EMU10K1
18: 7685557 IO-APIC-fasteoi ohci1394, eth1, nvidia
19: 3 IO-APIC-fasteoi ohci1394
20: 5721787 IO-APIC-fasteoi ehci_hcd:usb1
21: 291891 IO-APIC-fasteoi libata
22: 391482 IO-APIC-fasteoi libata
23: 14994686 IO-APIC-fasteoi ohci_hcd:usb2, eth0
NMI: 29848
LOC: 123722840
ERR: 0
|
This is bad:
17: 248525 IO-APIC-fasteoi libata, EMU10K1
18: 7685557 IO-APIC-fasteoi ohci1394, eth1, nvidia
The IRQ sharing may be my problem in the post here.
Hopefully this can be resolved.
Cheers,
QuantumSummers
_________________
Summers
"...there are no rules here -- we're trying to accomplish something."
- Thomas A. Edison |
|
| Back to top |
|
|
quantumsummers
Retired Dev
Joined: 24 Sep 2006
Posts: 103
Location: Earth
|
| Posted: Mon Jun 25, 2007 5:44 pm Post subject: possibly some interesting info |
|
|
Hi,
I found a _long_ file on this page that discussed ACPI & its relation to IRQs. Particularly, a tool called acpidump was mentioned in section 3.4 that may be useful in the current endeavor. Its in portage with a suite called sys-power/pmtools.
Lets see if this sheds any light on the issue.
Regards,
QuantumSummers
_________________
Summers
"...there are no rules here -- we're trying to accomplish something."
- Thomas A. Edison |
|
| Back to top |
|
|
mikers456
n00b
Joined: 07 Jul 2004
Posts: 36
|
| Posted: Mon Jul 02, 2007 5:00 am Post subject: |
|
|
I've been concerned about the same issue for some time now. I would really like to assign IRQs to eliminate sharing of big users like nvidia, eth*, and libata. Check this out:
My guess would be that the sharing there between nvidia and whatever else is a problem. Besides degraded performance overall when doing lots of stuff on any one of those devices.
I don't know if the status of video cards has changed in the last year or two but the advice I've read from just about _everywhere_ wrt to IRQs and video cards is DON'T SHARE. The problem is that video cards don't play nice with other devices and have some weird interrupt behavior that can cause a lot of grief. Particularly if expected to play nice with stuff like ethernet cards or ata controllers.
I'll see if I can dig up some links that give better or more detailed info besides hearsay.
... [edit]
As promised: Gentoo stability / hardware guide...
http://www.gentoo.org/doc/en/articles/hardware-stability-p2.xml
Thread on spurious interrupts from a nvidia GeForce card
http://www.nvnews.net/vbulletin/showthread.php?t=65617
Thread were the above one came from
http://www.nvnews.net/vbulletin/showthread.php?t=65621
Interrupt handling nitty-gritty, again -- sharing interrupts only works nice when all drivers are _well written_
http://www.evilbitz.com/2006/12/08/interrupts-and-interrupt-controllers/ |
|
| Back to top |
|
|
|
Gentoo on AMD64
