As with many Data Centers we’ve been deploying a large number of blade servers and switches. We’re primarily an HP shop from a server,desktop and laptop perspective and we’ve been working with HP Blade System for the past two years.
HP actually OEM’s two different GbE2 switches for their Blade enclosures. One is based off a Nortel (Blade Technologies) solution and the other is based off a Cisco solution. We’re using the Nortel version and we’ll be focusing on that hardware in this post. If your unfamiliar with the Alteon CLI your going to need a few minutes to catch on. It’s pretty simple but very different from either the Nortel CLI or the Cisco CLI. Another important point is that the enclosure can actually accommodate two HP GbE2 switches. There’s an “A” side and a “B” side. You only need an “A” side switch to provide connectivity for the servers that will be housed in the enclosure but for high-availability solutions you’ll definitely need two switches installed into the enclosure.
In the following post I’ll outline how to configure a HP GbE2 Switch Blade trunking both ports into a MultiLink trunk. We’ll only using one HP GbE2 switch for this example and ignore the “B” switch. You should console up to the HP GbE2 using a serial cable (straight thru cable 9600, 8, N 1). The default password is “admin”.
[Main Menu] Jul 19 8:07:04 NOTICE mgmt: admin login from host 10.101.20.1 info - Information Menu stats - Statistics Menu cfg - Configuration Menu oper - Operations Command Menu boot - Boot Options Menu maint - Maintenance Menu diff - Show pending config changes [global command] apply - Apply pending config changes [global command] save - Save updated config to FLASH [global command] revert - Revert pending or applied changes [global command] exit - Exit [global command, always available] >> Main#
Set Admin password
We’ll start out by setting the administrator password on the switch.
>> Main# /cfg/sys/access/user/admpw Changing ADMINISTRATOR password; validation required: Enter current admin password: Enter new administrator password: Re-enter new administrator password: New administrator password accepted.
Set IP Address
Next we’ll setup an IP address on one of the interfaces.
>> Main# cfg/l3/if 1 [IP Interface 1 Menu] addr - Set IP address mask - Set subnet mask vlan - Set VLAN number relay - Enable/disable BOOTP relay ena - Enable IP interface dis - Disable IP interface del - Delete IP interface cur - Display current interface configuration >> IP Interface 1#>> addr 10.101.255.118 Current IP address: 0.0.0.0 New pending IP address: 10.101.255.118 Pending new subnet mask: 255.0.0.0 >> IP Interface 1# mask 255.255.255.0 Current subnet mask: 0.0.0.0 New pending subnet mask: 255.255.255.0 >> IP Interface 1# vlan 200 Current VLAN: 1 New pending VLAN: 200 >> IP Interface 1# ena Current status: disabled New status: enabled
Set IP Default Gateway
Next we’ll setup a default gateway for the switch.
>> Main# cfg/l3/gw 1 [Default gateway 1 Menu] addr - Set IP address intr - Set interval between ping attempts retry - Set number of failed attempts to declare gateway DOWN arp - Enable/disable ARP only health checks ena - Enable default gateway dis - Disable default gateway del - Delete default gateway cur - Display current default gateway configuration >> Default gateway 1# addr 10.101.255.1 Current IP address: 0.0.0.0 New pending IP address: 10.101.255.1 >> Default gateway 1# ena Current status: disabled New status: enabled
Create Trunk Interface
We’ll create a Multilink trunk interface (Etherchannel) utilizing ports 19 and 20. Switch ports 19-22 are GBIC interfaces which are populated by 1000BaseSX SFPs.
>> Main# /cfg/l2/trunk 1 [Trunk group 2 Menu] add - Add port to trunk group rem - Remove port from trunk group ena - Enable trunk group dis - Disable trunk group del - Delete trunk group cur - Display current Trunk Group configuration >> Trunk group 2# add 19 Port 19 added. >> Trunk group 2# add 20 Port 20 added. >> Trunk group 2# ena Current status: disabled New status: enabled
Enable 802.1q (tagging) on fiber uplinks
The external uplinks are ports 19 and 20. The internal crossconnect links between the two HP GbE2 switches are on ports 17 and 18. We need to enable 802.1q VLAN tagging on the uplink ports so we can bridge multiple VLANs across the uplinks.
>> Main# /cfg/port 17 ------------------------------------------------------------ [Port 19 Menu] gig - Gig Phy Menu aclqos - Acl/Qos Configuration Menu 8021ppri - Set default 802.1p priority pvid - Set default port VLAN id name - Set port name rmon - Enable/Disable RMON for port tag - Enable/disable VLAN tagging for port tagpvid - Enable/disable tagging on pvid brate - Set BroadCast Threshold mrate - Set MultiCast Threshold drate - Set Dest. Lookup Fail Threshold ena - Enable port dis - Disable port cur - Display current port configuration >> Port 17# tag e Current VLAN tag support: disabled New VLAN tag support: enabled Port 17 changed to tagged.
I’m going to just provide the commands for the remaining ports and skip showing the enter text of the menu to help save on the length of this document.
>> Port 17# /cfg/port 18/tag e Current VLAN tag support: disabled New VLAN tag support: enabled Port 18 changed to tagged. >> Port 19# /cfg/port 19/tag e Current VLAN tag support: disabled New VLAN tag support: enabled Port 19 changed to tagged. >> Port 19# /cfg/port 20/tag e Current VLAN tag support: disabled New VLAN tag support: enabled Port 20 changed to tagged.
Create VLAN 200 for management of the switch itself.
>> Main# /cfg/l2/vlan 200 VLAN number 200 with name "VLAN 200" created. ------------------------------------------------------------ [VLAN 200 Menu] name - Set VLAN name stg - Assign VLAN to a Spanning Tree Group add - Add port to VLAN rem - Remove port from VLAN def - Define VLAN as list of ports ena - Enable VLAN dis - Disable VLAN del - Delete VLAN cur - Display current VLAN configuration >> VLAN 200# name "10-101-255-0/24” Current VLAN name: New VLAN name: 10-101-255-0/24 >> VLAN 200# add 17 Current ports for VLAN 200: empty Pending new ports for VLAN 200: 17 >> VLAN 200# add 18 Current ports for VLAN 200: empty Pending new ports for VLAN 200: 17-18 >> VLAN 200# add 19 Current ports for VLAN 200: empty Pending new ports for VLAN 200: 17-19 >> VLAN 200# add 20 Current ports for VLAN 200: empty Pending new ports for VLAN 200: 17-20
Spanning Tree Protocol (Disable STP on trunk uplinks)
>> Main# /cfg/l2/stp 1 ------------------------------------------------------------ [Spanning Tree Group 1 Menu] brg - Bridge parameter menu port - Port parameter menu add - Add VLAN(s) to Spanning Tree Group remove - Remove VLAN(s) from Spanning Tree Group clear - Remove all VLANs from Spanning Tree Group on - Globally turn Spanning Tree ON off - Globally turn Spanning Tree OFF default - Default Spanning Tree and Member parameters cur - Display current bridge parameters >> Spanning Tree Group 1# port 19 ------------------------------------------------------------ [Spanning Tree Port 19 Menu] prior - Set port Priority (0-255) cost - Set port Path Cost (1-65535 (802.1d) / 1-200000000 (MSTP/RSTP) / 0 for auto) link - Set port link type (auto, p2p, or shared; default: auto) edge - Enable/disable edge portfastfwd - Enable/disable Port Fast Forwarding modeon - Turn port's Spanning Tree ON off - Turn port's Spanning Tree OFF cur - Display current port Spanning Tree parameters >> Spanning Tree Port 19# off Current Port 19 Spanning Tree setting: ON New Port 19 Spanning Tree setting: OFF >> Main# /cfg/l2/stp 1/port 20/off Current Port 20 Spanning Tree setting: ON New Port 20 Spanning Tree setting: OFF
Network Time Protocol
>> Main# /cfg/sys/ntp ---------------------------------------------------------- [NTP Server Menu] prisrv - Set primary NTP server address secsrv - Set secondary NTP server address intrval - Set NTP server resync intervalt zone - Set NTP timezone offset from GMT dlight - Enable/Disable daylight savings time on - Turn NTP service ON off - Turn NTP service OFF cur - Display current NTP configuration >> NTP Server# prisrv 10.101.20.1 Current NTP server address: 0.0.0.0 Enter new NTP server address: 10.101.20.1 >> NTP Server# secsrv 10.111.20.1 Current NTP server address: 0.0.0.0 Enter new NTP server address: 10.111.20.1 >> NTP Server# tzone -5:00 Current GMT timezone offset: -8:00 Enter new GMT timezone offset in hours [-12:00, +12:00]: -5:00 >> NTP Server# on Current status: OFFNew status: ON >> NTP Server# dlight e Current status: disabledNew status: enable
Set PVID on Uplink Ports
>> Main# /cfg/port 17/pvid 200 >> Main# /cfg/port 18/pvid 200 >> Main# /cfg/port 19/pvid 200 >> Main# /cfg/port 20/pvid 200You might also want to read these other posts...
I have a question for you about your setup. I have am having some problems don’t know yet if its because of my config. But basically I have 2 ERS8600’s and 2 HP GbE2c’s . 4 fibers, 2 from the A and 2 from B. Blade SW 1 gets a A and B, same goes for SW 2. Both are MLT’ed and on the 8600. SW 1 has its own SMLT and the other has its own SMLT. I get some performance issues on one application. Is it the way I have it setup is why I see the server retransmitting. Oh BTW the server has automatic teaming set which it usually picks NFT.
Do I have to do VLACP on the 8600 and LACP on the switches to fix this?
Thanks!
Hi Hackez,
If you want to connect both HP GbE2c switches to your core network you’ll need to run Spanning Tree between your core and the two GbE2cs. The HP GbE2c’s don’t support IST/SMLT thats a Nortel proprietary protocol although I sometimes wonder why they don’t license it to Blade Technologies. You realize that there is a virtual trunk interface between the two switch (side/switch A and side/switch B) within the enclosure. If you want to take advantage of the HP NIC teaming and fault tolerance you’ll need run Spanning Tree. Another option would be to disable the virtual trunk interface (I believe it’s trunk 1 in the configuration by default) and then uplink both HP GbE2c’s individually to your core network. Then there would be no need for Spanning Tree.
Good Luck!
I currently don’t have any spanning tree as I run SMLT’s. I know the new switches support stacking. But the ones we have don’t. I know about port 17 and 18 and I have them disabled. BTW this is the current setup I have. http://img18.imageshack.us/img18/663/networklayout1.jpg
http://yfrog.com/07networklayout1xj
Isn’t a diagram worth a thousand words… :)
So in this configuration you’ll only be able to use NIC teaming in a fault tolerant configuration… essentially the second NIC will run in standby mode and will only come active if the primary NIC looses link with the network switch. If you had physical servers that were cabled directly to the ERS 8600 switch then you could configure a SMLT using LACP where you could use NIC teaming in an Active/Active configuration. Unfortunately since the HP GbE2c doesn’t support IST/SMLT you can only run them in NFT mode.
Hopefully I’ve made that clear…
Hehe Yes it is. Yup. It took me a week to figure out a save config. And that’s what I came up with, but what do you think on the current config? I’m worried somewhere between the GbE2c switches to the passports it may cause network problems.
That configuration is fine… you may want to explore UFD (Uplink Failure Detection) on the HP GbE2 switch. It will automatically disable the server ports if it looses uplink to your core causing the NIC teaming solution to fail-over to the redundancy standby NIC.
Good Luck!
safe*
I though my config was alright. I do have UFD enabled and its pretty cool. Thanks for the help!
I have a 300 with 8 servers in it. (1/2 height)
I want to put 1 server on our Local Lan and the other 7 on our DMZ.
I have configured the GBE2C card so it has 2 VLANS, and have added port 24 to VLAN2(for the DMZ), and connected an uplink cable into port 24 on the Interconnect bay 1.
(I think that is right so far).
Now I need to re-map NIC1 of each of the 7 servers to port 24.
Can you assist, or point me to a document that explains this.
(I am very new to this)
thanks in advance for any help.
Hi Steve,
You’re just about there… you really only need to add the specific edge (server) ports to your “DMZ” VLAN.
I’ll provide a little more explanation but essentially svl0r has provided you the commands in his response (interesting site).
I don’t know that I would advocate disabling Spanning Tree globally because it can lead to unexpected results. I would probably advise that you disable Spanning Tree per port.
The switch port on the HP GbE2c corresponds to the server slot on the HP 7000 Enclosure. There is a document from HP that documents the mapping which can get really confusing when you need more than 2 NICs and multiple HBAs.
NIC 1 on the server in slot 1 corresponds to port 1 on the A side of the GbE2 switch.
NIC 2 on the server in slot 1 corresponds to port 1 on the B side of the GbE2 switch
NIC 1 on the server in slot 2 corresponds to port 2 on the A side of the GbE2 switch
NIC 2 on the server in slot 2 corresponds to port 2 on the B side of the GbE2 switch
…repeat 16 servers (half-height) correspond to port 1-16…
Good Luck!
Steve,
http://pastebin.com/f11270546
Nice work… interesting site.
Cheers!
Yeah I use it on IRC channels to paste configs.