Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
bonding
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
Vieri
l33t
l33t


Joined: 18 Dec 2005
Posts: 870

PostPosted: Fri Jan 25, 2013 10:47 am    Post subject: bonding Reply with quote

Hi,

I'm trying to configure bonded interfaces between 2 Gentoo hosts.

Gentoo box 1 has 2 network interfaces, the first (eth2 in my example) is connected to port 1 on a 802.3ad switch and the other (eth3) to port 1 on another 802.3 switch.

Gentoo box 2 also has 2 NICs, one (eth2) connected to switch 1 on port 2 and the other to switch 2 on port 2.

Switch 1 and 2 are stacked.
Switch 1 port 1 and switch 2 port 1 are configured in the switch web UI as LACP.
Same for switch 1 port 2 and switch 2 port 2.

On Gentoo box 1:

Code:

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
 
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
 
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 301
        Partner Mac Address: fc:75:16:6a:a9:e1
 
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:11:62:4f
Aggregator ID: 1
Slave queue ID: 0
 
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:11:5e:77
Aggregator ID: 1
Slave queue ID: 0


Gentoo box 2 has a similar config but with different MAC addresses. However the "Partner Mac Address" is the SAME on both Gentoo servers.
Is this expected?

Also, I must be doing something wrong either on the switch or on the Gentoo boxes because disconnection of a cable "fails over" fine but if all cables are connected I'm expecting to have double speed (eg. 1Gbps * 2) but it's not the case.
I'm using D-LINK DGS-3100 switches and I'm thinking that maybe I'm configuring it inappropriately.
Any hints/ideas as to what I may check on the Gentoo servers and if anyone has successfully used DLINK switches for this kind of setup?

Thanks,

Vieri
Back to top
View user's profile Send private message
massimo
Veteran
Veteran


Joined: 22 Jun 2003
Posts: 1226

PostPosted: Fri Jan 25, 2013 11:20 am    Post subject: Reply with quote

What about your xmit_hash_policy? How do you test the throughput?
_________________
Hello 911? How are you?
Back to top
View user's profile Send private message
Vieri
l33t
l33t


Joined: 18 Dec 2005
Posts: 870

PostPosted: Fri Jan 25, 2013 2:21 pm    Post subject: Reply with quote

I use gkrellm to monitor both servers (http://members.dslextreme.com/users/billw/gkrellm/gkrellm.html).

/proc/net/bonding/bond0 shows that Transmit Hash Policy: layer2.
Same on the switch: layer2

On the DLINK switch the trunking is as follows:
ID Type Ports
01 LACP 1:1, 2:1
02 LACP 1:2, 2:2

So trunk Id 01 has LACP on switch 1, port 1 and switch 2, port 1
trunk Id 02 has LACP on switch 1, port 2 and switch 2, port 2

Note: "Partner Mac Address" is the MAC addr. of the switch so that should be fine.
Back to top
View user's profile Send private message
your_WooDness
Tux's lil' helper
Tux's lil' helper


Joined: 25 Oct 2007
Posts: 77

PostPosted: Sat Jan 26, 2013 10:52 am    Post subject: Reply with quote

Hi,

trunking / bonding normally only uses the aggregated bandwidth when the bond get's more connections. So with xmit_hash_policy layer2, a connection between the peers is setup through one interface, due to the MAC base selection of the path.
I think xmit_hash_policy layer3+4 will try to distribute the traffic on all slaves of the bond for a single connection, but it won't use the full available bandwidth due to fragmentation. And when I got it right, this policy is not 802.3ad compliant and you could experience problems...

You also could test the balanced-alb mode, which does a load balancing. Some call this mode "have-your-self-fun-mode" because you might also experience problems with that. I have it running at two NFS servers and haven't experienced any problems yet and performance is quiet good.

w00d
Back to top
View user's profile Send private message
Vieri
l33t
l33t


Joined: 18 Dec 2005
Posts: 870

PostPosted: Sat Jan 26, 2013 3:40 pm    Post subject: Reply with quote

your_WooDness wrote:

trunking / bonding normally only uses the aggregated bandwidth when the bond get's more connections. So with xmit_hash_policy layer2, a connection between the peers is setup through one interface, due to the MAC base selection of the path.


That's what I thought so I opened 3 sessions on 1 server and did an rsync of a huge file in each session towards the other server. So I had 3 different rsync processes from server 1 to server2.
Still, I had all the traffic going on just one interface. If I ifdown one of the 2 NICs or disconnect a cable then all traffic fails over correctly to the other interface (at the same transfer rate). But there's no way I can "load balance" between the two.

your_WooDness wrote:

I think xmit_hash_policy layer3+4 will try to distribute the traffic on all slaves of the bond for a single connection, but it won't use the full available bandwidth due to fragmentation. And when I got it right, this policy is not 802.3ad compliant and you could experience problems...


I also tried layer3+4 but didn't notice any difference compared to layer2. (same rates and same fail-over behavior with no load balancing)
That's why I'm starting to think that there may be something wrong or weird with my stacked switches.
How did you configure your switches?

your_WooDness wrote:

You also could test the balanced-alb mode, which does a load balancing. Some call this mode "have-your-self-fun-mode" because you might also experience problems with that. I have it running at two NFS servers and haven't experienced any problems yet and performance is quiet good.


I might as well give it a shot.

Thanks
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum