HOWTO: Bond two NICs together and connect to a Cisco switch

[tam]

I use an Intel Dual Server NIC, 2* 100MBit and a Cisco 2950 48 port Switch and of course gentoo Linux with kernel 2.6

This kind of bonding provides load balancing and fault tolerance.

1. Compile the bonding driver as module. The advatage is, you can give some paramters to the driver.
Device Driver -> Networking Support -> <M> Bonding driver support
Compile your kernel

Code: Select all

cd /usr/src/linux
make && make modules_install

2. To autoload the driver put the line

Code: Select all

bonding miimon=100 mode=0

into /etc/modules.autoload.d/kernel-2.6

3. Edit /etc/conf.d/net . For a fixed IP, mine looks like the following. If you look at the

Code: Select all

iface_bond0="192.168.14.5 netmask 255.255.255.0 broadcast 192.168.14.255 up"
gateway="bond0/192.168.14.210"

There is no entry for eth0 or eth1!

4. To install ifenslave.c, do

Code: Select all

cd /usr/src/linux/Documentation/networking
gcc -Wall -Wstrict-prototypes -O -I/usr/src/linux/include ifenslave.c -o ifenslave 
cp ifenslave /sbin/ifenslave

5. Edit the file /etc/conf.d/local.start to enslave the two NICs

Code: Select all

ifenslave bond0 eth0 eth1

6. make a symlink for bond0

Code: Select all

ln -sf /etc/init.d/net.eth0 /etc/init.d/net.bond0

7. Make sure you don't start eth0 or eth1, but start bond0

Code: Select all

rc-update del eth0
rc-update del eth1
rc-update add bond0 default

8.Reboot.

9. After reboot ifconfig should look similar to this. If you look at the RX and TX values of eth0 and eth1, you can see a nice load balancing.

Code: Select all

bond0     Link encap:Ethernet  HWaddr 00:11:22:33:44:55  
          inet addr:192.168.14.5  Bcast:192.168.14.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:833 errors:0 dropped:0 overruns:0 frame:0
          TX packets:226 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:360503 (352.0 Kb)  TX bytes:16845 (16.4 Kb)

eth0      Link encap:Ethernet  HWaddr 00:11:22:33:44:55  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:181991 (177.7 Kb)  TX bytes:8241 (8.0 Kb)

eth1      Link encap:Ethernet  HWaddr 00:11:22:33:44:55  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:413 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:178512 (174.3 Kb)  TX bytes:8604 (8.4 Kb)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

10. To configure EtherChannel use telnet to the Cisco. I have bonded port 23 and 24 together:

Code: Select all

Password:
cata2950>enable
Password:
cata2950#


cata2950#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
cata2950(config)#interface range FastEthernet0/23 -24
cata2950(config-if-range)#channel-group 1 mode auto
Creating a port-channel interface Port-channel1
cata2950(config-if-range)#end

cata2950#show running-config
[...]
interface FastEthernet0/23
 no ip address
 channel-group 1 mode auto
!
interface FastEthernet0/24
 no ip address
 channel-group 1 mode auto
!
[...]

cata2950#copy running-config startup-config
Destination filename [startup-config]?
Building configuration...
[OK]
cata2950#exit

For more info about the EtherChannel configuration go to the Cisco website

For more info about the bonding please read /usr/src/linux/Documentation/networking/bonding.txt

Fell free to add stuff or correct me.

[mkoroschetz]

The step by step instructions are great and easy to understand and follow.

Unfortunately in my setup I am getting the error when executing: ifenslave bond0 eth0

Master 'bond0', Slave 'eth0': Error: Enslave failed

and the only thing that happens is that "bond0" takes the HW address of the slave "(eth0) but there is no entry for eth0 in ifconfig and obviously it does not work.

Code: Select all

bond0     Link encap:Ethernet  HWaddr [b]08:00:20:c5:7a:15[/b]
          inet addr:192.168.1.250  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Environment:

HW platform: SPARC
Kernel: Gentoo - Linux version 2.4.26-sparc-r2
Bonding: Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
--> Compiled as module.
--> I am autloading de module with /etc/modules.autoload.d/kernel-2.4 (same as in step 2. above)
NIC: eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:c5:7a:15
--> Compiled into the Kernel
Tested ifenslave.c which installed with the Kernel, as well as the latest avialable: ifenslave-1.1.0-r1

Any suggestions will be greatly appreciated. Thanks in advance.

[tam]

Master 'bond0', Slave 'eth0': Error: Enslave failed

If you only have one NIC, you don't need the bonding driver.

[mkoroschetz]

Sorry for the misleading information . I am aware that it makes not much sense to do bonding with only one interface. The command that I need to work is really:

Code: Select all

ifenslave bond0 eth0 eth1

but I get the same error message:

Code: Select all

Master 'bond0', Slave 'eth0': Error: Enslave failed
Master 'bond0', Slave 'eth1': Error: Enslave failed

I don't see any other error message reported in the log files.

On the other hand a bond has to work with just one interface, because in case of a failure, that is exactly what could happen.

Thanks

[tam]

Di you bring up eth0 or eth1 before bond0? If so don't do this. I will add this to the guide.

See new 7. in the guide.

[mkoroschetz]

No, I did not bring up either eth0 or eth1 before bond0. There are no entries for eth0 and eth1 in /etc/init.d/net.

On this line of thinking the only thing to mention would be, that I have the driver for eth0 and eth1 compilled into the kernel. This means that the drivers get loaded before the bonding module.

By the way I am doing this on a sparc system, and eth0 and eth1 are SUNHME cards.

[tam]

On this line of thinking the only thing to mention would be, that I have the driver for eth0 and eth1 compilled into the kernel. This means that the drivers get loaded before the bonding module.

I have also compiled the driver for the NIC(s) into the kernel.

I don't know anbything about Sparc systems.

[Zigen]

Would it be possible to bond 4 NICS together?

1x Integrated NIC
1x Dual NIC Card
1x Single NIC card

[tam]

Would it be possible to bond 4 NICS together?

Sure. The only thing you have to change is

Code: Select all

ifenslave bond0 eth0 eth1 eth2 eth3

[Zigen]

Will this work if I have 3x 100 mbit and 1 x gigabit NIC ports on my server?

[tam]

Will this work if I have 3x 100 mbit and 1 x gigabit NIC ports on my server?

I did not try this, but there nothing in the docs that this won't work, so I think it should work. Pls try it and report here.

[Otm]

Will this work if I have no Cisco but i have 3 ISP connection (ADSL) ?

[jdgill0]

Will this work if I have no Cisco but i have 3 ISP connection (ADSL) ?

The help within menuconfig for bonding says the following

If you have two Ethernet connections to some other computer, you can make them behave like one double speed connection using this driver. Naturally, this has to be supported at the other end as well, either with a similar Bonding Linux driver, a Cisco 5500 switch or a SunTrunking SunSoft driver.

So it seems bonding is dependent upon the router having this capability or the computer at the other end having the bonding support in its kernel. Obviously, with your question you don't have control over the computer at the other end.

If this will not work for you, I am also in the process of trying something else that is similar. I read a website that talked about load balancing with multiple nics using a "virtual ip address" for all the nics. It did not explain in detail unfortunately. What I have done is assign the same IP address to ALL the nics in our master node (we are attempting to use a cluster and bandwidth is killing us). I don't know if what I have done will even work, as I made the changes remotely from home last night. Hopefully I have a chance to go into the lab today and try this out and do some more research about virtual ip addresses for multiple nics.

I also plan to try the bonding technique. Our cluster is using the school's switches, so I am unsure whether or not the client nodes will need bonding support in their kernel.

[ichoes]

The step by step instructions are great and easy to understand and follow.

Unfortunately in my setup I am getting the error when executing: ifenslave bond0 eth0

Master 'bond0', Slave 'eth0': Error: Enslave failed

and the only thing that happens is that "bond0" takes the HW address of the slave "(eth0) but there is no entry for eth0 in ifconfig and obviously it does not work.

Code: Select all

bond0     Link encap:Ethernet  HWaddr [b]08:00:20:c5:7a:15[/b]
          inet addr:192.168.1.250  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Environment:

HW platform: SPARC
Kernel: Gentoo - Linux version 2.4.26-sparc-r2
Bonding: Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)
--> Compiled as module.
--> I am autloading de module with /etc/modules.autoload.d/kernel-2.4 (same as in step 2. above)
NIC: eth0: HAPPY MEAL (PCI/CheerIO) 10/100BaseT Ethernet 08:00:20:c5:7a:15
--> Compiled into the Kernel
Tested ifenslave.c which installed with the Kernel, as well as the latest avialable: ifenslave-1.1.0-r1

Any suggestions will be greatly appreciated. Thanks in advance.

I dont know if you already solved your problem but the thing is with sun sparc is that they use the same MAC Address for all cards. You have to change the setting in the eprom to use their unique MAC address. I had to do it because in the university were i was working one of the routers was complaining that i was using the same ip addres for to NIC. After i changed the settings of the NIC everything went back to normal.

[xcorpse]

i have tried to use the sun qfe's with the older bonding driver (circa 2.4.x), however at the time the driver (sunqe) did not use the mii interface (which the bonding driver relied on for link status information). this requirement has changed in the kernel 2.6.x bonding driver, it now uses netif_carrier_ok by default so cards that are not mii aware can be used.

[mattsk]

I've got a similar setup happening. However, in my case it was important that the bonding module be brought up before any of the ethernet modules. I also had the problem that the eth1394 module would sometimes load itself before the module for my ethernet ports, meaning that sometimes I had to execute

Code: Select all

/sbin/ifenslave bond0 eth0 eth1

and sometimes

Code: Select all

/sbin/ifenslave bond0 eth1 eth2

I solved this with an entry in /etc/modules.d/ called bond (which was put there by the ifenslave ebuild, I think ... it seemed easier to emerge ifenslave than to do the above).

Code: Select all

# read /usr/src/linux/Documentation/networking/bonding.txt for help!

alias bond0 bonding
options bond0 mode=802.3ad miimon=100

alias eth0 tg3
alias eth1 tg3

# Make sure bond0 is initialised before the tg3 module.
below tg3 bond0

# Make sure our network interfaces are brought up before the firewire ethernet
pre-install eth1394 /sbin/modprobe eth0

#alias bond1 bonding
#options bond1 -o bonding1 arp_interval=200 arp_ip_target=10.0.0.1

# Parameters:
# max_bonds int, description "Max number of bonded devices"
# miimon int, description "Link check interval in milliseconds"
# use_carrier int, description "Use netif_carrier_ok (vs MII ioctls) in miimon; 0 for off, 1 for on (default)"
# mode string, description "Mode of operation : 0 for round robin, 1 for active-backup, 2 for xor"
# arp_interval int, description "arp interval in milliseconds"
# arp_ip_target string array (min = 1, max = 16), description "arp targets in n.n.n.n form"
# updelay int, description "Delay before considering link up, in milliseconds"
# downdelay int, description "Delay before considering link down, in milliseconds"
# primary string, description "Primary network device to use"
# multicast string, description "Mode for multicast support : 0 for none, 1 for active slave, 2 for all slaves (default)"
# lacp_rate string, description "LACPDU tx rate to request from 802.3ad partner (slow/fast)"

I deliberately used pre-install (instead of below) for eth1394 since the use of the below directive will cause the bdlow modules also be removed after (as well as installed before) the module in question. In this case, if I remove tg3 with modprobe -r it will also remove bonding, just as if I installed tg3 with modprobe, it would first install bond0. I could probably have gotten away with pre-install for this just as well.

Any comments as to the correctness of this module setup would be welcomed. I'd *REALLY* like some mechanism in the /etc/conf.d/net file to name eth0 and eth1 as the slaves of bond0, rather than having to frag around with the local.start file.

[briansrapier]

I realize that this is an older post, but more relavent than most others that I have come across. I am running Gentoo 2005.0, 2.6.11-gentoo-r6 kernel built from vanilla-sources with bonding compiled as a module. My issue is not with bonding starting up; it does that just fine:

bond0 Link encap:Ethernet HWaddr 00:11:09:B7:93:6A
inet addr:10.0.0.1 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:354065 errors:0 dropped:0 overruns:0 frame:0
TX packets:962 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:41367676 (39.4 Mb) TX bytes:289828 (283.0 Kb)

eth0 Link encap:Ethernet HWaddr 00:11:09:B7:93:6A
inet6 addr: fe80::211:9ff:feb7:936a/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:354065 errors:0 dropped:0 overruns:0 frame:0
TX packets:963 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41367676 (39.4 Mb) TX bytes:289986 (283.1 Kb)
Base address:0x9000 Memory:fb100000-fb120000

eth1 Link encap:Ethernet HWaddr 00:11:09:B7:93:6A
inet6 addr: fe80::211:9ff:feb7:936a/64 Scope:Link
UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Base address:0xa000 Memory:fb000000-fb020000

I have the following options set in modules.d/bond:

alias bond0 bonding
options bond0 mode=1 miimon=100 updelay=100 downdelay=100 arp_interval=200 arp_target=10.0.0.1 primary=eth0

The issue that I have is with failover from primary to secondary. When I 'hard fail' eth0 by pulling the connection or shutting down the interface, bonding fails to roll over to the second interface. However, if I plug the connection back in within a minute (+/-), the connection on eth0 remains active, which wasn't the case before I implemented bonding.

If I manually failover to the second NIC using ifenslave, I get:

root@host>ifenslave -vc bond0 eth1
ifenslave.c:v1.1.0 (December 1, 2003)
o Donald Becker (becker@cesdis.gsfc.nasa.gov).
o Detach support added on 2000/10/02 by Willy Tarreau (willy at meta-x.org).
o 2.4 kernel support added on 2001/02/16 by Chad N. Tindel
(ctindel at ieee dot org).
ABI ver is 2
Slave eth1: flags 1803.
Master 'bond0': Error: SIOCBONDCHANGEACTIVE failed: Invalid argument
Master 'bond0', Slave 'eth1': Error: Change active failed

Currently, the server in question is connected, both eth0 and eth1, to a D-Link managed switch. Once put into production, the server will be directly connected to an ImageStream Linux router. Could either of these devices be causing my problems?

[Ausdonky]

Hi brian..

looks like your switch might not be setup correctly.. can you supply a model number for the one you are using?? Does it have 802.3ad trunking support?? (Not all D-link managed switches have 802.3ad support!)

wrt the ImageStream server.. if the problem is that your switch does not have trunking support the linux server should be ok as long as you setup bonding on that machine also..

HTH

Andrew

[smile032]

bond0 has already been configured for eth3 and eth4, and all is running well. Then I try to bond eth1 and eth2. I edit /etc/conf.d/net and comment eth1 and eth2 configuration, and add a new line iface_bond1="bla bla bla" to set the bond1 IP. I do cp /etc/init.d/net.bond0 /etc/init.d/net.bond1, and then try to run the net.bond1 interface. What I found is "no device found". I think that bond1 device is not found by the kernel. How can I add the device bond1? I believe that I should modprobe a module to solve this problem.. Any idea ??

Thanks in advance

[gnork]

Add the following to /etc/modules/d/bond (created by "emerge ifenslave")

Code: Select all

alias bond1 bonding 

For further configuration have a look at /usr/src/linux/Documentation/networking/bonding.txt
There are some issues with the loading sequence of the modules, so I really suggest you read that first.

HTH
gnork