Joined: 15 Jun 2006
Location: Budapest, Hungary
|Posted: Mon Jun 19, 2006 2:51 pm Post subject: TC, bridge and SMP problem
I'm using 2.6.16-gentoo-r9 kernel on a dual-Xeon box. There is 3 NIC's in the server. The first (eth0) one is a 3c59x. This is for management reasons. The second and the third (eth1, eth2) is both e1000 cards. They are connected as a bridge (br0). I set up an iptables filter for MAC<->IP authorization on the br0 and rate limiting on the eth1 and eth2.
It works fine. My problem is that all interrupt handled by the first CPU only and in the high traffic times it can be the bottleneck ( all of the CPU time is spent for getting packets from the NIC's ).
I installed irqbalance. When I started it the load suddenly goes to extremely high. I had to reboot. Ok. Maybe this is a bug in irqbalance. So I tried to set it with "echo 02 >/proc/irq/217/smp_affinity" ( 217 is the IRQ for eth1 ). The load goes to skies until I didn't stop the rate limiting. Then the load was okay, but the "echo 01 >/proc/irq/217/smp_affinity" didn't worked so I rebooted again....
What can be the problem? This rate limiting worked on a Fedora with the same hardware using irqbalance. The differences is the kernel, the tc and the Fedora box didn't had bridge. It routed the packets before the rate limiting.
I want to migrato to Gentoo all of our core systems and it's a very big problem for me....
Some diagnostic output:
Gentoo Base System version 1.6.14
Linux xxxxxxxxx 2.6.16-gentoo-r9 #5 SMP PREEMPT Mon Jun 12 14:59:54 CEST 2006 i686 Intel(R) Xeon(TM) CPU 3.20GHz GNU/Linux
tc utility, iproute2-ss060110
00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub (rev 0c)
00:00.1 Class ff00: Intel Corporation E7320 Error Reporting Registers (rev 0c)
00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 0c)
00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A1 (rev 0c)
00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge (rev 02)
00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal Host Controller (rev 02)
00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal Host Controller (rev 02)
00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer (rev 02)
00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable Interrupt Controller (rev 02)
00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced Host Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a)
00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface Controller (rev 02)
00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02)
03:03.0 Ethernet controller: Intel Corporation 82545GM Gigabit Ethernet Controller (rev 04)
04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
04:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller (rev 05)
04:05.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 30)
0: 420362591 482 IO-APIC-edge timer
2: 0 0 XT-PIC cascade
8: 1 1 IO-APIC-edge rtc
14: 2 2 IO-APIC-edge libata
15: 146069 4 IO-APIC-edge libata
169: 1699411977 1 IO-APIC-level uhci_hcd:usb2, eth2
193: 0 0 IO-APIC-level uhci_hcd:usb3
201: 2637189 1 IO-APIC-level eth0
209: 0 0 IO-APIC-level ehci_hcd:usb1
217: 1610187401 1 IO-APIC-level eth1
NMI: 0 0
LOC: 420357798 420357797
Any suggestion are welcome!
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum