I recently needed to look at HP’s NIC Teaming feature within their Proliant server product line to provide “ultra” high-availability to a new Windows 2008 Cluster which was running Microsoft Exchange 2007. It seems that NIC teaming has come a long ways from the original NIC fault tolerance where one NIC would simply act as a standby should the first loose LINK with the network switch.
HP’s NIC Teaming now supports 802.3ad (LACP) which provides a load balanced and fault tolerant solution. I’ve posted several documents to my website for anyone that might be looking for additional information.
Here is a very good document that outlines NIC teaming from head to toe, along with additional documents that outline the different capabilities and what operating systems are supported.
In short you need to setup an LACP group on the Nortel switch for any ports that you want to be in the LAG (Link Aggregation Group). You can leave the HP NIC teaming options set to Automatic and it should negotiate as a “802.3ad Dynamic with Fault Tolerance” when you have the Nortel switch configuration properly.
5520-48T-PWR> enable 5520-48T-PWR# config term 5520-48T-PWR(config)# interface FastEthernet 1/1-2 5520-48T-PWR(config-if)#lacp aggregation port 1/1-2 enable 5520-48T-PWR(config-if)#lacp mode port 1/1-2 active 5520-48T-PWR(config-if)#lacp key port 1/1-2 101
Cheers!
The link http://www.michaelfmcnmara.com/files/TeamingWP.pdf
is broken…
Thanks for the heads up Tom. I’ve fixed the link.
i wonder if servers-connected ERS switch (not sure if it’s 8600 or stacking family) has a feature that can shutdown access ports when uplink connection fails. it’s not SMLT, but more like black-hole prevention.
e.g.:
– ERS switches below is a server farm switch
– if ERS-A detects failure in uplink ports to L2-switch A, to prevent blackholing traffic coming from servers, the ERS shuts down the access ports (where servers are attached to).
– so, automatically, servers will switchover to the other NIC to use ERS-B.
++++++++++
| L2-switch A |
++++++++++
||
|| uplink
||
+————–+
| |
| ERS A |
| |
| |
| access |
+————–+
| | | |
| | | |
| | | |
server farms (dual home/NIC-team?)
| | | |
| | | |
| | | |
+————–+
| |
| ERS B |
| |
| |
| access |
+————–+
||
|| uplink
||
++++++++++
| L2-switch |
| |
++++++++++
thanks.
Hi Lou,
You bring up some really good stuff. I believe you’re making reference to UFD (Uplink Failure Detection) which is present in some of the HP GbE2 switches and HP ProCurve switches.
You might also want to look at VLACP (Virtual Link Aggregation Control Protocol). It’s a proprietary extension of LACP that Nortel uses to provide a “hearbeat” if you will between two switches. It’s primarily used when connecting to end points over an intermediate network (such as a carriers Ethernet backbone). However it plays a very important role in how redundant networks detect and recover from a network failures.
So NIC teaming with LACP should provide you that hearbeat between the servers and the access switch. You would then run VLACP across an SMLT uplink to two core ERS 8600 switches. There is also an advanced license key available from HP that enables SMLT link functionality on the HP NIC teaming side of things although I’ve never investigated that.
You can find additional information on VLACP from this post.
On another note I’ve never run a LAG (802.3ad) in an SMLT configuration. We primarily use Nortel’s MultiLink trunking (MLT) which is proprietary to Nortel equipment but I’m guessing it’s probably supported.
Thanks for the comment!
HP Discontinued the Advanced / Inteligent Networking Pack option late 2008, so you no longer have the Dual Channel options.
We currently have some of our Servers configured in SLB mode, and using Dual 8300 Nortel cores (SMLT/IST/etc) We have it split accross the cores using S-SMLT (Single Port SMLT)
You also need to turn off Spanning tree completly on these ports in order for it to work.
This did not work on the 3.0.3 Codebase, and we upgraded to 4.1.1.0 to fix alot of other issues too. (MS- NLB Support, etc)
It provides us redundant cores, full speed, and active failover. It works great.
Hi Bill,
Thanks for the feedback and information!
Thanks for the comment!
Hello Michael;
I have searched high/low for the exact steps to utilize LACP on a Nortel 8310 using 8348GTX cards, and am unsuccessful thus far. Below is what I have.
* server with dual nics
* Admin wants us to config LACP on our Nortel ERS 8310 chassis running 4.1.1 code.
* I will utilize 2 different 8348GTX modules.
Here is what I have configured so far, but it doesn’t appear to want to play nicely.
2 PORTS on 8310 configured with:
* lacp ENABLED
* key 5
* aggregation TRUE
* mode ACTIVE
* member of server vlan
MLT 5
*created mlt 5
* issued mlt 5 lacp enable
* issued mlt 5 to use key 5
* LACP enabled.
* aggregatable.
* added the appropriate VLAN the server should use.
**UNABLE to add port members w/o getting the following error:
“rcMLTPortMemebers.5:This MLT is used for link aggregation.”
This is my first time setting one of these up, I must be doing something wrong.
Any thoughts/guidance would be greatly appreciated.
Thanks in advance,
aaron
Hi Aaron,
You’ve actually done too much configuration. You don’t need to configure a Multi-Link trunk, you only need to enable LACP on the specific ports and set them to the same key value. A LAG (Link Aggregation Group) is the same thing as an MLT (Multi-Link Trunk) only first is dynamic while the second is statically configured. I don’t have specific experience on the ERS 8300 but I would assume that it works just the same as the ERS 5500 or ERS 8600 switches.
I’ve posted the Link Aggregation Control Protocol(LACP) 802.3ad and VLACP for ES and ERS Technical Configuration Guide on my website. I would suggest you review the document if you have any questions about LACP and/or VLACP.
Please let us know if that solves your problem.
Good Luck!
Michael,
thanks for the quick response, and for sharing the information you did. (I read through the documentation, and made some changes to my original config.) We didn’t get a chance to test this configuration against a server yet, but plugged an ERS 5520 (configured for LACP) into the 8310 LACP enabled ports and did notice port assignments/vlan under the MLT tab dynamically populate, so it appears to be working.
Again, many thanks.
aaorn
Michael,
you were absolutely right, I was killing it with “over configuration.” The simplified “right” configuration was tested and works like a champ.
thanks again, and have a productive week.
aaron
Dear MIchael,
I am trying to use LACP for some HITACHI NAS servers they support link aggregation 802.3ad dynamic. What is the signifance of Key? as on my servers i dont have any option to put this key. The only option i get is link aggregation add servers NIC to it and assign IP to them. Can i use default key?
Thanks
ZUbair
Hi Zubair,
You won’t have any option on the client side of things (Hitachi NAS). You will have options on the network switch. The admin key helps the switch to identify which ports should be grouped (trunked) together. If the ports you are using in connecting the NAS have different key values then the network switch will not know it should trunk them together.
Good Luck!
Hi Michael,
I am having problems trying to SMLT to Linux-based devices. All of them support 802.3ad link aggregation / bonding. I think the problem might be on my Nortel end. I have 2x 8310 chassi and I need to allocate 1 port from each chassi and connect it to a Linux router (Vyatta, etc). I have the bonding configured on the Linux box. I was wondering what would be the configuration on the Nortel end.
I tried going into the port and set SLT, but apparently it didnt work. I was only able to Ping one of the switches when using single port SMLT. I tried LACP and it didnt work either. I must be doing something wrong. Can you show me/correct me in terms of the Nortel configuration? Someone mentioned about using LACP via SMLTSystemID (basically LACP on SMLT) but I am kinda confused as SMLT ports cannot be used for LACP (the JVM does not permit it).
Thanks in advance,
Nicholas
Thanks Michael , its 2nd time your blog helps me . I didnt knew i need to configure ports in order to work with server aggregation nics , evryone i asked told me i need to configure nothing on switch )-: . Now i know why my netapp and backup server are working slow.
Can you make configuration example for 8600?
Hi Michael,
i am trying to configure lacp btw cisco 3020 blade enclosure switch and ERS8600. there are 4 links in the trunk with auto-negotiate enabled. one link is negotiating at 100mbps and not becoming part od lacp. on cisco side he port is showing as suspended.
8600 config,
# MLT CONFIGURATION
#
mlt 24 create
mlt 24 ntstg disable
mlt 24 lacp enable
mlt 24 lacp key 3
# LACP CONFIGURATIO
#
config lacp enable
# PORT CONFIGURATION – PHASE I
#
ethernet 2/19 perform-tagging enable
ethernet 2/20 perform-tagging enable
ethernet 2/21 perform-tagging enable
ethernet 2/22 perform-tagging enable
# STG CONFIGURATION
#
stg 1 remove ports 2/23
stg 1 add ports 2/20-2/22,2/37-2/39,4/41-4/48
# PORT CONFIGURATION – PHASE II
#
ethernet 2/19 default-vlan-id 501
ethernet 2/19 stg 1 stp disable
ethernet 2/19 lacp key 3
ethernet 2/19 lacp aggregation true
ethernet 2/19 lacp timeout short
ethernet 2/19 lacp enable
ethernet 2/20 default-vlan-id 501
ethernet 2/20 stg 1 stp disable
ethernet 2/20 lacp key 3
ethernet 2/20 lacp aggregation true
ethernet 2/20 lacp timeout short
ethernet 2/20 lacp enable
ethernet 2/21 default-vlan-id 501
ethernet 2/21 stg 1 stp disable
ethernet 2/21 lacp key 3
ethernet 2/21 lacp aggregation true
ethernet 2/21 lacp timeout short
ethernet 2/21 lacp enable
ethernet 2/22 default-vlan-id 501
ethernet 2/22 stg 1 stp disable
ethernet 2/22 lacp key 3
ethernet 2/22 lacp aggregation true
ethernet 2/22 lacp timeout short
ethernet 2/22 lacp enable
What is Cisco Side configuration? can you post here. Also can you explain a little, all these ports get suspeneded or not.
only one port 2/21(nortel side)–> 0/19(cisco side) got suspended.
!
spanning-tree mode pvst
spanning-tree extend system-id
interface Port-channel10
switchport trunk encapsulation dot1q
switchport trunk native vlan 501
switchport mode trunk
!
interface GigabitEthernet0/17
switchport trunk encapsulation dot1q
switchport trunk native vlan 501
switchport mode trunk
media-type rj45
speed 100
duplex full
channel-protocol lacp
channel-group 10 mode active
!
interface GigabitEthernet0/18
switchport trunk encapsulation dot1q
switchport trunk native vlan 501
switchport mode trunk
media-type rj45
speed 100
duplex full
channel-protocol lacp
channel-group 10 mode active
!
interface GigabitEthernet0/19
switchport trunk encapsulation dot1q
switchport trunk native vlan 501
switchport mode trunk
media-type rj45
speed 100
duplex full
channel-protocol lacp
channel-group 10 mode active
!
interface GigabitEthernet0/20
switchport trunk encapsulation dot1q
switchport trunk native vlan 501
switchport mode trunk
media-type rj45
speed 100
duplex full
channel-protocol lacp
channel-group 10 mode active
!
Hi Matt,
suspecting something related to port 21(nortel side) harsdware, i tried changing the port to 18(nortel side) but no luck. this time port 20(nortel side) gone out.
below are the show cmd outputs,
DataCenter_Core1:5# sh mlt info
================================================================================
Mlt Info
================================================================================
PORT SVLAN MLT MLT PORT VLAN
MLTID IFINDEX NAME TYPE TYPE ADMIN CURRENT MEMBERS IDS
——————————————————————————–
24 4119 MLT-24 trunk normal norm norm 2/18-2/19,2/22 501 506
28 4123 Hitachi_NAS access normal norm norm 2/37-2/39 503
29 4124 SMLT-MC-Srv access normal smlt norm 4/41 501
30 4125 SMLT-DC-DIST access normal smlt smlt 4/42-4/45 13
32 4127 DC-SW-IST trunk normal ist ist 4/46-4/48 12 13 50
1 502 503 504 505 506
MULTICAST DESIGNATED LACP LACP
MLTID IFINDEX DISTRIBUTION NT-STG PORTS ADMIN OPER
——————————————————————————–
24 4119 disable disable 2/18 enable up
28 4123 disable enable 2/37 enable up
29 4124 disable enable null disable down
30 4125 disable enable 4/42 disable down
32 4127 disable enable 4/46 disable down
DataCenter_Core1:5# sh mlt lacp info 4119
================================================================================
LACP Aggrgator Information
================================================================================
MAC COLLECTOR AGGR PORT
MLTID IFINDEX ADDR MAXDELAY ORINDI MEMBERS
——————————————————————————–
24 4119 00:1c:9c:1f:90:51 32768 aggr 2/18-2/19,2/22
——————————————————————————–
OPER OPERLAST
MLTID IFINDEX STATE CHANGE
——————————————————————————–
24 4119 up 304 day(s), 14:16:07
——————————————————————————–
ACTOR ACTOR ACTOR ACTOR
MLTID IFINDEX SYSPRIO SYSID ADMINKEY OPERKEY
——————————————————————————–
24 4119 32768 00:1c:9c:1f:90:00 3 3
——————————————————————————–
PARTNER PARTNER PARTNER
MLTID IFINDEX SYSPRIO SYSID OPERKEY
——————————————————————————–
24 4119 32768 00:24:c3:93:ee:80 10
DataCenter_Core1:5# sh port info lacp all port 2/18,2/19,2/20,2/22
================================================================================
Actor Admin
================================================================================
INDEX SYS SYS KEY PORT PORT STATE
PRIO ID PRIO
——————————————————————————–
2/18 32768 00:1c:9c:1f:90:00 3 0x91 32768 act long aggr
2/19 32768 00:1c:9c:1f:90:00 3 0x92 32768 act long aggr
2/20 32768 00:1c:9c:1f:90:00 3 0x93 32768 act long aggr
2/22 32768 00:1c:9c:1f:90:00 3 0x95 32768 act long aggr
================================================================================
Actor Oper
================================================================================
INDEX KEY SELECTED ATTACHED AGGR STATE
AGGR ID AGGR ID
——————————————————————————–
2/18 3 4119 4119 true act long aggr sync col dis
2/19 3 4119 4119 true act long aggr sync col dis
2/20 1171 147 147 true act long indi sync def
2/22 3 4119 4119 true act long aggr sync col dis
================================================================================
Partner Admin
================================================================================
INDEX SYS SYS KEY PORT PORT STATE
PRIO ID PRIO
——————————————————————————–
2/18 0 00:00:00:00:00:00 0 0x0 0 pas long indi
2/19 0 00:00:00:00:00:00 0 0x0 0 pas long indi
2/20 0 00:00:00:00:00:00 0 0x0 0 pas long indi
2/22 0 00:00:00:00:00:00 0 0x0 0 pas long indi
================================================================================
Partner Operational
================================================================================
INDEX SYS SYS KEY PORT PORT STATE
PRIO ID PRIO
——————————————————————————–
2/18 32768 00:24:c3:93:ee:80 10 0x11 32768 act long aggr sync col
dis
2/19 32768 00:24:c3:93:ee:80 10 0x12 32768 act long aggr sync col
dis
2/20 0 00:00:00:00:00:00 0 0x0 0 pas long indi
2/22 32768 00:24:c3:93:ee:80 10 0x14 32768 act long aggr sync col
dis
================================================================================
LACP Extention
================================================================================
INDEX ADMIN OPER FAST SLOW AGGRWAIT TIMEOUT FAST SLOW AGGRWAIT TIME
OUT
ENABLED ENABLED TIME TIME TIME SCALE TIME TIME TIME SCALE
ADMIN ADMIN ADMIN ADMIN OPER OPER OPER OPER
——————————————————————————–
2/18 true true 1000 30000 2000 3 1000 30000 2000 3
2/19 true true 1000 30000 2000 3 1000 30000 2000 3
2/20 true false 1000 30000 2000 3 1000 30000 2000 3
2/22 true true 1000 30000 2000 3 1000 30000 2000 3
when i just forced the port speed to 100mbps full duplex on both sides of the link, all the ports got bundled in the lacp. when i revert back to auto-negotiation, only three links got bundled at 1gig and the fourth link negotiated at only 100mbps.
HI Siraj,
Are you using a cross-over cable or does the HP 3120 Interconnect provide MDI-X features? What card are you using in the Nortel ERS 8600? 8648GTR, 8648GTRS?
As you already know your configuration is right on the money… very interesting that you have duplex mis-match issues with that new equipment.
Thanks for the comment!
PS: I’m not sure that Cisco supports short timers with LACP?
can you try by changing all trunk ports to access as I think there is only one vlan in it.
Also try what Michael has written.
I think we have been fed some duff advice on HP NIC teaming! We use HP proliants plugged into two 8600’s with high availability. We were told to create an SMLT (across the two 8600’s)and then manualy configure the HP NIC teaming software to use “Switch Assisted Load Balancing and Fault tolerance” on “TCP”
Is this wrong?
Hi Dave,
I believe this is the old school (static) way of configuring a trunk using Nortel’s proprietary MLT feature. While this configuration will obviously work there are some drawbacks. The biggest draw back that stands out to me is the lack of any failure detection beyond link loss. Using the new LACP 802.3ad configuration allows both sides to dynamically add/remove links and allows both sides the ability to poll the far end providing a greater deal of failure detection beyond just link loss.
The LACP feature has really only matured in the past few years so prior to that you had Nortel’s MLT and Cisco’s Etherchannel.
Thanks for the comment!
I really appreciate you taking time to respond to this Michael. This site is excellent and a veritable treasure chest of useful information. I must take more time to help out and comment on here!
Hi,
In Nortel 8310 switch one of the port get shutdown. What should i troubleshoot?
Hi Raj,
I would suggest you post your issues in the discussion forums
You’ll probably want to provide some additional information for anyone to offer any advice and/or help.
Good Luck!
Hi,
I Need to install new Cisco L2-2960 with Nortel L3 8310 switch.
please tell me how can i connect both switch and what config
i should fellow on both switch’s So Nortel L3 pass vlan information with Cisco L2-2960 without any issue.
Thanks
Parvesh Katoch
Micheal,
Thanks for all the information it is very helpful. I need to connect a server to a pair of 8600 chassis’s that are connected together with IST’s. The question I have is would this be a simple MLT with LACP configuration or do I need to configure it as SMLT?
The configuration I have compiled for MLT with LACP is below.
ERS8600-A
config ethernet 1/44
config vlan 1 ports remove 1/44
config vlan 2 ports add 1/44
ERS8600-B
config ethernet 1/44
config vlan 1 ports remove 1/44
config vlan 2 ports add 1/44
ERS8600-A
config ethernet 1/44 lacp key 2
config ethernet 1/44 lacp aggregation true
config ethernet 1/44 lacp timeout short
config ethernet 1/44 lacp enable
ERS8600-B
config ethernet 1/44 lacp key 2
config ethernet 1/44 lacp aggregation true
config ethernet 1/44 lacp timeout short
config ethernet 1/44 lacp enable
ERS8600-A
config mlt 2 create
config mlt 2 name “MLT_LACP_1”
config mlt 2 lacp key 2
config mlt 2 lacp enable
ERS8600-B
config mlt 2 create
config mlt 2 name “MLT_LACP_1”
config mlt 2 lacp key 2
config mlt 2 lacp enable
Hi Jeff,
If you want to connect your server to multiple switches you need to utilize SMLT.
You need to add the following command to your configuration.
config mlt 2 smlt create smlt-id 2
Good Luck!
Micheal,
Thanks again for the assistance. I found an Avaya document – Data Center Server Access Solution Guide
NN48500-577, Date: July 2010, Version: 1.1 – They also state that it needs to be SMLT, however in the configuration they present they don’t use LACP, the configuration example is the same for both 8600’s:
config mlt 2 create
config mlt 2 perform-tagging enable
config mlt 2 add port 1/44
config vlan 2 add-mlt 2
config mlt 2 smlt create smlt-id 2
Choose switch assisted server load balancing with failover protection on the HP server.
Do you agree with this or should I stick with the MLT\LACP configuration as presented earlier and just add SMLT as you recommended. The Avaya configuration looks simplier on the surface, however that doesn’t always mean it will work better.
Thanks
Jeff
Hi Jeff,
If your equipment supports LACP (802.3ad) you want to use that over just a static port channel, Ether channel, Multi-Link trunk, etc. LACP will provide a heartbeat mechanism and several other features that regular static grouping/bonding doesn’t support. It will also facilitate better control over the load-balancing mechanism that’s employed to distribute the traffic across the multiple paths.
Why don’t you test it yourself ahead of time?
I have run several similar tests using an HP DL360 G5 creating an HP NIC team leaving the NIC teaming software configured as “Automatic” (default). The server will look for the LACP keepalives and will automatically configured itself for LACP/802.3ad mode if it sees those LACP packets. If it doesn’t see any LACP packets it will fall back to NFT (Network Fault Tolerant), perhaps even NFT with TX load-balancing if your NIC software supports that.
Good Luck!
Michael,
Thanks again for the input. I will proceed with the LACP config and test it out. Appreciate getting your real world thoughts compared to the Avaya default config.
Later
Jeff
It’s no problem at all… you might want to browse the forums;
http://forums.networkinfrastructure.info/
You better yet post your findings so that others following in your footsteps can benefit.
Cheers!
Hello,
I am trying to setup a Nortel 3510-24T for teaming. I have 2 servers connected, a Windows 2008 Server and an Openfiler SAN that I connect via iSCSI. Both boxes have Intel dual port 82576 nics. This is a test system so I am evaluating teaming between the two boxes and speed improvements.
When I follow your example above I get an error on line 5. I am not sure if the command is valid for this particular switch. This is all new to me and my first attempt at teaming so please excuse my lack of knowledge.
3510-24T>enable
3510-24T#config term
Enter configuration commands, one per line. End with CNTL/Z.
3510-24T(config)#interface fastethernet 1/11-12
3510-24T(config-if)#lacp aggregation port 1/11-12
^
% Invalid input detected at ‘^’ marker.
3510-24T(config-if)#lacp aggregation port 1/11-12 enable
Not sure what I am doing wrong.
Todd
^
Hi Todd,
I don’t believe the 3500 series supports LACP so the most you can probably get is a NFT (Network/NIC Fault Tolerant) configuration where one NIC is active and the other is passive.
Cheers!
Hi Mike,
Thanks for the response. The specs say it does LACP but I am not sure what method. The web menu has Multilink Trunking. I have read the manual and it seem pretty straight forward but as soon as I enable the MLT the ports loose connectivity or it becomes “flakey”: ping will work one-way or from some machines but not from others. The switch seems to work fine in every other respect.
I may just see if I can find another switch to work with. I had this one and it works like a champ on as individual ports. I can saturate the full gig and it runs fine. I was just hoping to go beyond that.
Do you know where I might be able to find the the current firmware?
Anyway I appreciate you taking the time to respond on a 3yr old post :)
Todd
Hi Todd,
I believe every Nortel (now Avaya) switch supports MultiLink Trunking (Nortel/Avaya proprietary trunking) but not every switch supports LACP (industry standard trunking).
You would probably be better served by configuring a second IP interface on your Openfiler and configure some of your client to utilize that IP address. That way you can control which clients are using which NIC.
Cheers!
Michael,
You are correct. After doing more reading I found LACP was planned for a future release of the software. I configured the 2 Intel NICSs on the Win28k server for Static Link Aggregation and I can connect to the server now with MLT configured on the switch but I am not sure if I am getting the benefit of more than 1 gig bandwidth. Stats on the switch indicate that the majority of the traffic IN is through one nic and OUT is through the other. That may be on the Openfiler side as it doesnt have a option listed for SLA but test from multiple simultaneous sources are still only getting 1 gig throughput.
I am making headway! Its been a learning experience and that counts in this case more than the results.
Todd
Dear Michael,
Thanks for all your detailed explanations.
In my setup, I have HP C7000 with 16 Blades + 2 X HP GbE2c + 2 X PP 8800. As you know, in 16 Server configuration the NIC’s of each server connected different GbE2c switch port. my both passports are connected in Active-Active mode with IST.
I want to achieve load balancing and FT as 802.3ad does. I can connect 4ports from each 8800 to GbE2c in SMLT. for this I should disable 17 & 18 port of GbE2c. But without this interswitch link(17&18), how to achieve 802.3ad teaming.
Could you help me on the exact configuration, I need to on each switch.. ?
Thanks a lot
The HP GbE2c does not support 802.3ad between the different interconnects (switches). So you can’t have active/active Layer 2 pathing, you need to use Spanning Tree to block one of the paths.
You can disable the cross-links (ports 17/18) and run SMLT to each interconnect (switch) but you can still only run the servers in a NFT (Network Fault Tolerant) mode.
Good Luck!
Hi Michael,
Thanks for your response..
In NFT mode it works fine, thought of exploring 802.3ad.
Thanks once again, keep up the good work.