bonding

Message

Vieri · Post by **Vieri** » Fri Jan 25, 2013 10:47 am

Hi,

I'm trying to configure bonded interfaces between 2 Gentoo hosts.

Gentoo box 1 has 2 network interfaces, the first (eth2 in my example) is connected to port 1 on a 802.3ad switch and the other (eth3) to port 1 on another 802.3 switch.

Gentoo box 2 also has 2 NICs, one (eth2) connected to switch 1 on port 2 and the other to switch 2 on port 2.

Switch 1 and 2 are stacked.
Switch 1 port 1 and switch 2 port 1 are configured in the switch web UI as LACP.
Same for switch 1 port 2 and switch 2 port 2.

On Gentoo box 1:

Code: Select all

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
 
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
 
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 301
        Partner Mac Address: fc:75:16:6a:a9:e1
 
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:11:62:4f
Aggregator ID: 1
Slave queue ID: 0
 
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:11:5e:77
Aggregator ID: 1
Slave queue ID: 0

Gentoo box 2 has a similar config but with different MAC addresses. However the "Partner Mac Address" is the SAME on both Gentoo servers.
Is this expected?

Also, I must be doing something wrong either on the switch or on the Gentoo boxes because disconnection of a cable "fails over" fine but if all cables are connected I'm expecting to have double speed (eg. 1Gbps * 2) but it's not the case.
I'm using D-LINK DGS-3100 switches and I'm thinking that maybe I'm configuring it inappropriately.
Any hints/ideas as to what I may check on the Gentoo servers and if anyone has successfully used DLINK switches for this kind of setup?

Thanks,

Vieri

massimo · Post by **massimo** » Fri Jan 25, 2013 11:20 am

What about your xmit_hash_policy? How do you test the throughput?

Vieri · Post by **Vieri** » Fri Jan 25, 2013 2:21 pm

I use gkrellm to monitor both servers (http://members.dslextreme.com/users/bil ... rellm.html).

/proc/net/bonding/bond0 shows that Transmit Hash Policy: layer2.
Same on the switch: layer2

On the DLINK switch the trunking is as follows:
ID Type Ports
01 LACP 1:1, 2:1
02 LACP 1:2, 2:2

So trunk Id 01 has LACP on switch 1, port 1 and switch 2, port 1
trunk Id 02 has LACP on switch 1, port 2 and switch 2, port 2

Note: "Partner Mac Address" is the MAC addr. of the switch so that should be fine.

your_WooDness · Post by **your_WooDness** » Sat Jan 26, 2013 10:52 am

Hi,

trunking / bonding normally only uses the aggregated bandwidth when the bond get's more connections. So with xmit_hash_policy layer2, a connection between the peers is setup through one interface, due to the MAC base selection of the path.
I think xmit_hash_policy layer3+4 will try to distribute the traffic on all slaves of the bond for a single connection, but it won't use the full available bandwidth due to fragmentation. And when I got it right, this policy is not 802.3ad compliant and you could experience problems...

You also could test the balanced-alb mode, which does a load balancing. Some call this mode "have-your-self-fun-mode" because you might also experience problems with that. I have it running at two NFS servers and haven't experienced any problems yet and performance is quiet good.

w00d

Vieri · Post by **Vieri** » Sat Jan 26, 2013 3:40 pm

your_WooDness wrote: trunking / bonding normally only uses the aggregated bandwidth when the bond get's more connections. So with xmit_hash_policy layer2, a connection between the peers is setup through one interface, due to the MAC base selection of the path.

That's what I thought so I opened 3 sessions on 1 server and did an rsync of a huge file in each session towards the other server. So I had 3 different rsync processes from server 1 to server2.
Still, I had all the traffic going on just one interface. If I ifdown one of the 2 NICs or disconnect a cable then all traffic fails over correctly to the other interface (at the same transfer rate). But there's no way I can "load balance" between the two.

your_WooDness wrote: I think xmit_hash_policy layer3+4 will try to distribute the traffic on all slaves of the bond for a single connection, but it won't use the full available bandwidth due to fragmentation. And when I got it right, this policy is not 802.3ad compliant and you could experience problems...

I also tried layer3+4 but didn't notice any difference compared to layer2. (same rates and same fail-over behavior with no load balancing)
That's why I'm starting to think that there may be something wrong or weird with my stacked switches.
How did you configure your switches?

your_WooDness wrote: You also could test the balanced-alb mode, which does a load balancing. Some call this mode "have-your-self-fun-mode" because you might also experience problems with that. I have it running at two NFS servers and haven't experienced any problems yet and performance is quiet good.

I might as well give it a shot.

Thanks