Multiple NIC problems

Message

DeathCount · Post by **DeathCount** » Sat Feb 19, 2011 12:47 am

Hello everybody,

Having a few issues with my multi NIC server. I have three gigabit network ports (1x onboard and 2x on a dual port PCIe) and for some reason all three initialize like they should but only one ever gets used.
For example, my onboard nic always has all traffic go through it regardless of intended ip address. So no matter what ip I ping (192.168.1.25, 192.168.1.26, 192.168.1.27) it all goes through the onboard port.
I know this because when I unplug the cable from it, I am unable to ping any address even though the other two still have cables and lights on the router. I am using gentoo-sources-2.6.36-r5 with
udev-151-r4. Thanks for your help!

My onboard NIC: (r8169)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

My PCIe NICs: (e1000e)
02:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
02:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

ifconfig

Code: Select all

eth0      Link encap:Ethernet  HWaddr 00:22:xx:xx:9a:7e  
          inet addr:192.168.1.25  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:881 errors:0 dropped:0 overruns:0 frame:0
          TX packets:518 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:315062 (307.6 KiB)  TX bytes:54882 (53.5 KiB)
          Interrupt:42 Base address:0xa000 

eth1      Link encap:Ethernet  HWaddr 00:15:xx:xx:10:22  
          inet addr:192.168.1.26  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 KiB)  TX bytes:0 (0.0 KiB)
          Interrupt:18 Memory:fdfe0000-fe000000 

eth2      Link encap:Ethernet  HWaddr 00:15:xx:xx:10:23  
          inet addr:192.168.1.27  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 KiB)  TX bytes:0 (0.0 B)
          Interrupt:19 Memory:fdfa0000-fdfc0000

/etc/conf.d/net

Code: Select all

config_eth0=( "192.168.1.25 netmask 255.255.255.0" )
routes_eth0=( "default via 192.168.1.1" )

config_eth1=( "192.168.1.26 netmask 255.255.255.0" )
routes_eth1=( "default via 192.168.1.1" )

config_eth2=( "192.168.1.27 netmask 255.255.255.0" )
routes_eth2=( "default via 192.168.1.1" )

70-persistent-net.rules

Code: Select all

# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:22:xx:xx:9a:7e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x105e (e1000e)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:xx:xx:10:22", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x105e (e1000e)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:xx:xx:10:23", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

Kernel Config - Network Section

Code: Select all

#
# Networking options
# 
CONFIG_PACKET=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
CONFIG_IP_MROUTE=y
# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
# CONFIG_INET_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
# CONFIG_TCP_CONG_HSTCP is not set
# CONFIG_TCP_CONG_HYBLA is not set
# CONFIG_TCP_CONG_VEGAS is not set
# CONFIG_TCP_CONG_SCALABLE is not set
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
# CONFIG_IPV6 is not set
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y

Jaglover · Post by **Jaglover** » Sat Feb 19, 2011 1:00 am

You can't have multiple NICs on same subnet just like that.

http://www.google.com/linux?hl=en&safe= ... tnG=Search

your_WooDness · Post by **your_WooDness** » Sat Feb 19, 2011 1:06 am

Hi DeathCount,

this is because of the so called "arp flux". If you have several NICs in one subnet, the first NIC will answer on the MAC layer address for all addresses on the subnet. So all arp tables of switches and other servers will have only one MAC address for, in your case three IP addresses. You can check this by using arping from a host pingin the host with the three NICs. You will get one and the same MAC address for each IP address.

There are different ways to work around this. Not sure, but I think there is a option somewhere in /proc you can alter. Search for "arp flux". Best would be to configure a bonding on the NICs if you only use one subnet.

WooD

TJNII · Post by **TJNII** » Sat Feb 19, 2011 1:57 am

That config will never work the way you expect, regardless of ARP. Ask yourself these questions: What is my IP? What are my routes? What ethernet device is used for which subnet? What IP do other machines use to connect to me?

Ethernet bonding isn't going to work unless you have something on the other end which supports bonding. If you don't have a bonding capable switch, you can't share traffic across all 3 nics transparently. If you have a home switch, you're SOL. Buy a managed switch.

You can hack around this with routing, by having all local traffic go through eth0 and all the outside traffic go through eth1. That is just a suboptimal hack using 2 Nics, though.

Buy a bridge capable switch and set up link bonding. That will do what you want.

DeathCount · Post by **DeathCount** » Sun Feb 20, 2011 12:07 am

Thanks for the replies everybody. What about setting up bonding with mode 6? I hear that has the benefits of 802.3ad without special hardware. Am I incorrect?

TJNII · Post by **TJNII** » Sun Feb 20, 2011 2:04 am

AFAIK that isn't true bonding, then. With true bonding all the ports function as one with one MAC address. That is why you need a bonding capable switch, as a regular switch will be very confused by having 3 pots share a MAC address.

However, that is just semantics. If it will do what you want, go for it. Keep in mind that that won't load balance across multiple NICs on a per connection basis using that methodology, though. Traffic to one endpoint will go through one NIC, not all 3. However, traffic to 3 endpoints should use all 3 NICs.

TJNII · Post by **TJNII** » Sun Feb 20, 2011 2:52 am

Also, out of curiosity, what are you doing that requires so much chutzpah from your network? I've never been able to choke a 1 gig connection, I've always hit other throttles first.

DeathCount · Post by **DeathCount** » Tue Feb 22, 2011 1:34 am

I'm not actually hitting a bottleneck really but it's more for multi-tasking. I originally set this up like this for LAN parties. The onboard nic would dedicate to serving all game server traffic while the other 2 nics would serve patches, files, isos, etc. full speed to multiple users without affecting game server traffic. When I'm not having a LAN party, I still would want the onboard to continue to serve game server traffic for my online gaming community and other "server" stuff such as SSH while the other two would do file sharing and 1080p video streams to multiple devices throughout the house at the same time. So in reality, no real reason besides multi-tasking.
Same reason this server has a quad-core processor and 2 hard drives setup in raid0 for sharing.... No real reason besides multi-tasking, and the fact I can do it.

Thanks again for all your help everybody.