Problems with running 5+ nics [solved]

jav · n00b Joined: 09 Dec 2007 Posts: 3

Hello everyone!

I got a system which has been working fine with my beloved Tulip Card. A Dlink DFE-570TX REV-1A.

I recently received a second card, which also is a D-Link DFE570TX REV-1A.
However, when I have both connected at the same time, I the interfaces just won't receive an IP from DHCP.
Each card works fine if it is installed alone, or together with another (single nic) card. But the two D-Links just won't play nice together.

I cannot see any thing wrong except for that it will not bind an IP.

If I simply remove any of the two cards (and re-arrange /etc/udev/rules.d/70-persistent-net.rules so I eth's still make sense) it will immediately work fine.

I have considered the possibility that I have borked things up by inserting and removing the cards (e.g. so I end up with an eth47 or similar), however the /etc/udev/rules.d/70-persistent-net.rules claims that my ethX are bound to the MAC addresses I expect them to (and I'm not getting any error messages about the NIC missing or drivers missing).

Another peculiar thing is that if I run "halt -p" with only one of the cards in the box powersoff nicley, if I run it with both cards in it won't poweroff (only stop HDDs and halt the system, the CPU fan and other fans keep on going).

Does any one have a clue as where I can continue to debug?

(thank you firefox for the spell checker

)

EDIT: I have also tried using a livecd (gentoo 2.6.19-r5)

jav · n00b Joined: 09 Dec 2007 Posts: 3

It turns out that my motherboard, for some strange reason, didn't like me filling up the PCI slots.

I'm still unsure as to why, but by leaving one PCI slot empty everything now works fine.

I'll look around for BIOS upgrades and the like.

If any one has suggestions on how to pinpoint the exact problem, please let me know.
(e.g. stuff I should check out regarding the PCI slots and IRQs or whatever).

MostAwesomeDude · Guru Joined: 12 Aug 2007 Posts: 373

cynric · Posted: Mon Jan 14, 2008 7:09 pm Post subject:

I too have had this problem. The machine would hardlock with 3 NICs. After disabling all the unneeded services to free up IRQs and fiddled with IRQ sharing, I was able to get 4 NICs going. Still working on the 5th, but it doesn't look good. IBM has a pretty good article on hardware stability. Not overly helpful in this case though, but if you don't know what interrupts are being used by what this might be useful.
_________________
"This Snow Crash thing -- is it a virus, a drug, or a religion?" "What's the difference?"
-- Neal Stephenson - Snow Crash

Simba7 · Posted: Mon Jan 14, 2008 9:35 pm Post subject:

I have a total of 12 in mine (including an Atheros Super-G Card).

2x Digital DE504's (Quad Port 10/100Base-TX)
1x Allied Telesyn AT-2560FX (10/100Base-FX)
1x D-Link DGE-500T (10/100/1000Base-TX)
1x 3Com 3c509B (10Base-T)
1x Atheros AR5212 Super-G Wireless Card

No problems here. Running them all on a i440BX Board with a Pentium II @ 450MHz.

Cyker · Veteran Joined: 15 Jun 2006 Posts: 1427

I hate you

Does it 'splode when you run them all at full speed? :mrgreen:

Out of curiosity, what motherboard are you using? I didn't think the 440BX has such a stable (A?)PIC...!
I'm just surprised you don't have IRQ hell with that many cards!!

(Seriously, the IRQ routing in modern systems is abysmal... I almost miss the old DOS days where you could manually set IRQs instead of being forced to use ACPI-assigned IRQs which are all IRQ 10 or randomly based on the PCI slot you use or something!)

cynric · Posted: Tue Jan 15, 2008 4:02 am Post subject:

The control in the DOS days was nice. Obviously his motherboard is a help, but having different vendors for the cards and them being multiport probably helps considerably. At least in my case with Linksys wanting to hang on only a handful of IRQs.
_________________
"This Snow Crash thing -- is it a virus, a drug, or a religion?" "What's the difference?"
-- Neal Stephenson - Snow Crash

Cyker · Veteran Joined: 15 Jun 2006 Posts: 1427

Wish mine was like that... my mobo doesn't even respect IRQ assignments at all!

The two onboard NICs are a sky2 and a forcdeth - No way to change their IRQs in BIOS and apparently there just is no mechanism in the 2.6 kernels for manually setting IRQs.

I tried booting Windows off a USB key to set them, which worked, but when you boot back to Linux they end up at the same values they were before

There's also a 3Com 905b combo card in there, which has the EEPROM configured to use IRQ5, but again Linux ignores it and sets its own IRQ.

So now I have the bizzare situation of 2 1000BaseTX, a 10Base2 card and the OHCI USB controller sharing a single IRQ (16), the other APIC IRQs being used for EHCI and various SATA controllers, while almost all of the PIC (sub-15) IRQs totally unused! :shock:

This leads to interesting hangs if all the cards are run at full speed... (The buggy sky and 3com drivers don't help either...)

Darn you IRQ sharing! Darn you to heck!!

Simba7 · Posted: Tue Jan 15, 2008 3:39 pm Post subject:

Cyker · Veteran Joined: 15 Jun 2006 Posts: 1427

Nice... my 'board's an Asus A8N32-SLi.

It's pretty crap...