HP

LACP Configuration Examples (Part 3)

11
August 29, 2009
by Michael McNamara in , ,

In part 3 of this series I’ll provide a relatively simple example of a LACP LAG between a HP GbE2c L2/L3 switch and two Nortel switches, we’ll terminate two different LAGs on the two ERS 8600 switches using Nortel’s proprietary SMLT (Split MultiLink Trunking) technology.

Example 2 – Ethernet Routing Switch 8600 to a set of HP GbE2c L2/L3 switches using LACP trunks with SMLT

As I said before a picture is worth a thousand words and can be very helpful in designing any network topology.

lacp-example3

I’m going to skip the configuration of the two Nortel Ethernet Routing Switch 8600s since you can refer to the earlier post for an example of how to configure them. In this design we need to disable the virtual cross connect that exists between the A and B sides of the two HP GbE2c switches. Please note that I’m working with the HP GbE2c (C-Class enclosure) not the GbE2 (P-Class enclosure). There are some slight differences between the two. The virtual trunk ports between the A and B sides are on ports 17 and 18 so those ports need to be disabled in order to prevent a loop.

HP-GbE2c-A / HP-GbE2c-B
/c/port 17/dis
/c/port 18/dis

With the virtual trunk cross connects disabled we can now wire each switch independently to the upstream switch(s) which in this case happens to be two ERS 8600s. As is usual for me I’ll create a network management VLAN and place the IP interface of each GbE2c switch in that VLAN (VLAN 200).

HP-GbE2c-A / HP-GbE2c-B
/c/l2/vlan 200
/c/l2/vlan 200/ena
/c/l2/vlan 200/name "10-101-255-0/24"

Let’s add VLAN 200 to the two ports, 21 and 22, that we’ll be using to uplink to the 8600 switches. We haven’t yet enabled tagging so the switch will ask you if you’d like to change the PVID from VLAN 1 (default) to VLAN 200, you can safely answer yes to this question.

HP-GbE2c-A / HP-GbE2c-B
/c/l2/vlan 200/add 21
/c/l2/vlan 200/add 22

Let’s enable tagging on both uplink ports along with RMON and set the PVID just to be safe;

HP-GbE2c-A / HP-GbE2c-B
/c/port 21/tag ena
/c/port 21/pvid 200
/c/port 21/rmon e
/c/port 22/tag ena
/c/port 22/pvid 200
/c/port 22/rmon e

Let’s turn off Spanning Tree on the uplinks, we only want Spanning Tree local to the switch since SMLT will take care of providing the loop free topology.

HP-GbE2c-A / HP-GbE2c-B
/c/l2/stp 1/port 21/off
/c/l2/stp 1/port 22/off

Now it’s time to configure LACP and create the LAG (Link Aggregation Group). We’ll using LACP key 50 but you could use any admin key (number) so long as both ports are configured with the same admin key.

HP-GbE2c-A / HP-GbE2c-B
/c/l2/lacp/port 21/mode active
/c/l2/lacp/port 21/adminkey 50
/c/l2/lacp/port 22/mode active
/c/l2/lacp/port 22/adminkey 50

Here’s the special sauce that will work in combination with the NIC teaming software to fail over in the event of an upstream switch problem or an uplink problem where the GbE2c continues to function but there’s a problem upstream. This configuration will cause the GbE2c switch to disable (admin-down) the server switch ports in the event that the LACP group goes down. This will cause the NIC teaming configuration on the servers to fail-over to the standby NIC.

HP-GbE2c-A / HP-GbE2c-B
/c/ufd/on
/c/ufd/fdp/ltm/addkey 50
/c/ufd/fdp/ltd/addport  1
/c/ufd/fdp/ltd/addport  2
/c/ufd/fdp/ltd/addport  3
/c/ufd/fdp/ltd/addport  4
/c/ufd/fdp/ltd/addport  5
/c/ufd/fdp/ltd/addport  6
/c/ufd/fdp/ltd/addport  7
/c/ufd/fdp/ltd/addport  8
/c/ufd/fdp/ltd/addport  9
/c/ufd/fdp/ltd/addport 10
/c/ufd/fdp/ltd/addport 11
/c/ufd/fdp/ltd/addport 12
/c/ufd/fdp/ltd/addport 13
/c/ufd/fdp/ltd/addport 14
/c/ufd/fdp/ltd/addport 15
/c/ufd/fdp/ltd/addport 16

If you haven’t already let’s configure an IP address (for management) on VLAN 200;

HP-GbE2c-A
/c/l3/if 1/ena
/c/l3/if 1/addr 10.1.255.128
/c/l3/if 1/mask 255.255.255.0
/c/l3/if 1/broad 10.1.255.255
/c/l3/if 1/vlan 200

We need to use a different IP address for the B side switch on VLAN 200;

HP-GbE2c-B
/c/l3/if 1/ena
/c/l3/if 1/addr 10.1.255.129
/c/l3/if 1/mask 255.255.255.0
/c/l3/if 1/broad 10.1.255.255
/c/l3/if 1/vlan 200

As mentioned by a few other folks on this blog and in the forums this solution only provides an active/passive solution in terms of the NIC teaming configuration. This is because the GbE2c L2/L3 switches don’t support IST/SMLT technology. While this will only provide 1Gbps of bandwidth (2Gbps if you count full duplex) between the blade server and the network it will provide significant level of redundancy and high-availability. In this design the network is protected from a GbE2c switch failure, a Nortel Ethernet Routing Switch 8600 failure, and multiple uplink/downlink failures.

Please feel free to post comments and questions here about this post. Questions regarding specific configurations can be posted in the forums; http://forums.networkinfrastructure.info/nortel-ethernet-switching/

Cheers!

HP Virtual Connect Smart Link

7
August 5, 2009
by Michael McNamara in

hpweb_logoHP’s Virtual Connect supports a feature called Smart Link, a network enabled with Smart Link automatically drops link to the server ports if all uplink ports lose link. This feature is very similar to the Uplink Failure Detection (UFD) that is available on the HP GbE2, GbE2c and most ProCurve switches. I believe there is a similar feature available on Cisco switches called Link State Tracking.

You might be asking, so what. Well a reader recently mentioned Smart Link in conjunction with my post concerning HP Virtual Connect & vSphere 4 so I thought I’d post some of my thoughts about Smart Link. It’s seems redundant to me given the presence of LACP but let me explain.

From a Nortel perspective you can connect two NICs from a single server to a Nortel switch cluster, two ERS 5500 or 8600 switches, using LACP and get a  “802.3ad Dynamic with Fault Tolerance” configuration. This essentially provides an active/active solution utilizing both NICs to their fullest. LACP is used to determine network path failures.

From a Virtual Connect perspective the same applies as above. The Virtual Connect Ethernet interconnect modules act as a single switch fabric allowing you to create a “802.3ad Dynamic with Fault Tolerance” configuration providing an active/active solution. While you can do this to servers you can’t span external uplinks across interconnects out of the enclosure in an active/active configuration.

In the old days you’d only get a “Transmit Load Balancing with Fault Tolerance” or “Network Fault Tolerance” configuration when your server NICs spanned two switches. This essentially provided an active/standby solution. Network Fault Tolerance only uses link status as a determinator to whether the network is functional or not. In order to detect a failure the server would need to see a link loss on the primary NIC before failing over to the standby NIC. Smart Link provides the ability to shutdown the server switch ports if all the uplink switch ports go down so the NIC teaming configuration can detect the link status change on the server and fail-over to the standby NIC which would be cabled to a different network switch.

In this case it would appear to me that LACP has really replaced what I would describe as a legacy feature, Smart Link. You can have multiple external uplinks out of an enclosure spread across multiple interconnects. While only the external uplinks on a single interconnect will be active, any remaining uplinks on any other interconnects will be in a standby mode.

I’m telling this as I see it, I’m no expert concerning Virtual Connect by any means so please tell me if I’m wrong!

Thoughts?

References;
http://blog.michaelfmcnamara.com/2009/01/hp-nic-teaming-with-nortel-switches/
http://www.michaelfmcnamara.com/files/TeamingWP.pdf

HP Virtual Connect & VMware vSphere 4

2
August 2, 2009
by Michael McNamara in

HP 7000 EnclosureAs I’ve mentioned in the past we’re kicking off a very large endeavor to move a significant number of our servers to a virtual environment. Over the past two weeks we built out an HP 7000 enclosure with 4 HP BL460c server blades, 6 HP Virtual Connect 1/10 Gb-F Ethernet interconnects, and 2 HP Virtual Connect 8Gb 24-Port Fiber Channel interconnects. The purpose of this hardware is to provide a temporary staging location as we perform the physical to virtual conversions before moving the virtual machines (across the network) to a new data center where additional VMware vSphere 4 clusters will be waiting.

We had some small issues when we first turned up the enclosure but the biggest hurdle was our unfamiliarity with Virtual Connect and locating the default factory passwords (we had ordered the enclosure as a special build so it came pre-assembled which saved us a lot of time and effort and was well worth the small added cost).

We’re currently using two Nortel Ethernet Routing Switch 5530s in a stack configuration mounted at the top of the rack. We also have a Nortel Redundant Power Supply Unit (RPSU) 15 installed to provide redundant power in the event that we loose a one of the rooms UPS’s or that we have an issue with an internal power supply in either ERS 5530 switch.  We load software release 6.1 onto the ERS 5530s and so far haven’t observed any issues. We’re initially connecting the ERS 5530 stack via 2 1000BaseSX (1Gbps) uplinks distributed across both ERS 5530 switches (DMLT) to a pair of Nortel Ethernet Routing Switch 8600s running in a cluster configuration using IST/SMLT(SLT) trunking. As the solution grows we can expand the uplink capacity by adding additional 1Gbps uplinks or by installing 10Gbps XFPs. We’re downlinking from the ERS 5530 stack to multiple HP Virtual Connect 1/10Gb-F modules using LACP. Unfortunately you can’t have a LAG span multiple HP Virtual Connect 1/10Gb-F Ethernet modules as this time. If you do, only the ports on one of the modules will be “Active” while the ports on other modules will by in “Standby”.

hpvirtualconnect110gbfThe HP Virtual Connect 1/10 Gb-F Ethernet interconnects provide 16 Internal 1Gb Downlinks,  4 External 10/100/1000BASE-T Uplinks, 2 External 10/100/1000BASE-T SFP Uplinks, 2 External 10Gb XFP Uplinks, 1 External 10Gb CX-4 Uplinks, and 1 10Gb Internal Cross Connect. Using the internal 10Gbps cross connect along with the external 10Gb CX-4 uplink you can create a 10Gbps network within the enclosure. You can also link multiple enclosures together to form a 10Gbps network contained entirely within the rack. This could be very beneficial in keeping vMotion and other unneeded traffic off the core uplinks.

In testing we did run into a significant problem that already appears to have been documented by HP although a solution is yet to be formulated. In testing several failure scenarios (physically removing the HP Virtual Connect Ethernet interconnects or remotely powering them down) we observed a significant problem when the interconnects where restored. The HP Virtual connect 1/10Gb-F would show no link to the blade server while the VMware 4 console would indicate that there was link. This problem obviously affected all traffic associated with that port group. The solution was to either reboot the VMware host or reset the NIC using ethtool -r  {NIC} from the server console.

Here’s the excerpt from the release notes;

When a VC-Enet module is running VC v1.30 firmware or higher, a link might not be re-established
between the module and the ports of an NC364m mezzanine card under the following conditions:

  • The network mappings are changed on the NIC ports through Virtual Connect.
  • The VC-Enet module is restarted from a power-cycle or reboot, or the module is removed and inserted.

If the server is rebooted, the link is established on all ports on both sides of the connection. Manually
toggling the link from the server should also restore the link.

The jury is still out on HP’s Virtual Connect although I hope to dig deeper in later posts.

Cheers!

References;

http://h18004.www1.hp.com/products/blades/components/c-class-interconnects.html

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01730696/c01730696.pdf

Go to Top