Michael McNamara

technology, networking, virtualization and IP telephony

Short Story – switchport trunk allowed vlan

by

I like sharing these stories because they help me document some really simple problems that can sometimes take a few minutes to troubleshoot and ultimately resolve. The moral of this story resolves around the much used command “switchport trunk allowed vlan x,y,z” and the often overlooked commands “switchport trunk allowed vlan add|remove x,y,z“.

I decided to write about this topic since I recently encountered operational “difficulties” with both my prior employer and my current employer that involved the same near identical mistake of a network engineer accidentally overwriting the list of allowed VLANs. In the most recent case it was a simple oversight on the engineer’s part and the problem was quickly corrected. On the prior case the engineer had issued the command “no switchport trunk allowed vlan x” which seems to have given NXOS a bit of a fit. The ports that were in that vPC needed to be shutdown and then enabled to clear what appeared to be a software bug. While the running-config indicated that the VLANs were being trunked on the ports, the MAC/FDB table had no entries of those VLANs on the affected ports.

I strongly recommend that folks prune VLANs that aren’t being used from their trunks, however, you need to be very careful with how you add and/or remove VLANs from the list once the trunk is up and running.

In the past I’ve seen folks accidentally overwrite the VLAN allowed add list by using the “switchport trunk allowed vlan” command. Look at this sample configuration;

interface port-channel2
  description VPC_CISCO_NEXUS_5010
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 150-154
  switchport trunk allowed vlan add 155
  mtu 9216
  vpc 2

Now let’s say we wanted to add VLAN 156 and we using the following command, taking the allowed VLAN list and adding VLAN 156;

switchport trunk allowed vlan 152-154,156

The problem with this is that we just missed the fact that VLAN 155 was also on that trunk and we just removed it from the trunk with the that previous command.

The morale of the story – be careful when you add/remove VLANs from trunk ports and make sure you use the switchport trunk allow vlan add|remove command.

Cheers!

Image Credit: Roger Kirby

Secondary Data Center – Where have I been?

by

It was just over 2 years ago that I designed and stood up our first off-campus data center in Philadelphia, PA. Since that time we’ve completely vacated our original data center migrating all the servers, applications and services out to our new data center. Last month we relocated our offices leaving the old data center and office space behind forever. The new office space is very nice and has a lot of (very needed) conference rooms all of which have built-in audio/video capabilities with either an over-head projector or flat screen TV. I’m still hoping to have a LAN party someday on those 61″ monster displays perhaps with Call of Duty: Black Ops 2?

In June we started deploying our secondary data center with the intent of providing our own business continuity and disaster recovery services for our tier 1 applications including all our data storage needs. The design allows us the flexibility to utilize both DCs in an active/active configuration with the ability to move workloads (virtual machines) between DCs. While the design allows us that option we’re still testing how we’re going to handle all the different disaster scenarios – blade, enclosure, rack, SAN, cage, entire data center, etc. While our primary data center rings in at 800 sq ft our secondary data center is only 300 sq ft. This is possible because we’re utilizing a traditional disaster recovery model for our big box non-tier 1 applications that for one reason or another aren’t virtualized. This helps reduce the number of lazy assets hanging around and helps control some of the budget numbers. I totally expect the number of big box applications to continue to shrink over time as more and more application vendors embrace virtualization.

We’ve had pretty good success with the design of our first data center so we only made a few corrections. There’s a lot of logistics that need to be considered in any design especially around all the power and cooling requirements.

The Equipment

What equipment did we use? We already deployed Cisco at our primary data center so we decided to stay with Cisco at our secondary data center.

  • Cisco Nexus 7010
  • Cisco Nexus 5010
  • Cisco Nexus 2248
  • Cisco Nexus 1000V
  • Cisco Catalyst 3750X
  • Cisco Catalyst 2960G
  • Cisco ASA5520
  • Cisco ACE 4710
  • Cisco 3945 Router (Internet)
  • Cisco 2811 Router (internal T1 locations)

What racks did we use for the network equipment?

  • Liebert Knurr Racks
  • Liebert MPH/MPX PDUs

What equipment did we use for the servers/blades?

  • HP Rack 10000 G2
  • HP Rack PDU (AF503A)
  • HP IP KVM Console (AF601A)
  • HP BladeSystem C7000 Enclosure
  • HP Virtual Connect Flex-10 Interconnect
  • HP SAN 8Gb Interconnect
  • Cisco Catalyst 3120X
  • HP BL460c G7
  • HP BL620c G7
  • HP DL380 G8
  • HP DL360 G8

What are we using for storage?

  • IBM XIV System Storage Gen3 (SAN) (w/4 1Gbps iSCSI replication ports)
  • IBM SAN80B-4 SAN Switch
  • EMC DD860 (Disk-Disk backup via Symantec NetBackup)

Additional miscellaneous equipment;

  • MRV LX-4048T (terminal server)

We had some challenges with designing our secondary data center due to the density of our equipment. We had to stay under the maximum kw per sq foot load that the room (data center) was designed to handle. This is a simple calculation based on the kW utilization of the equipment to determine if there is adequate power and cooling available to meet that demand. We also had to maintain a N+1 design so we really can’t consuming more than 40% of our capacity leaving 10% for reserve. While some vendors charge a flat fee for the space (includes power) others charge per kWh so it’s very important to understand what type of demand you’re going to be placing on the data center.

My Design

We stood up a pair of Ciena 5200s from Zayo (formerly AboveNet) providing us a DWDM ring with 4 wavelengths between our primary data center and secondary data center . We’re using 2 wavelengths for the IP network between 2 pairs of Cisco Nexus 7010s and 2 wavelengths for the SAN fiber channel network between 2 pair of IBM SAN switches. We have the option of adding upwards of 4 additional wavelengths before we need to add any hardware so we have room for growth. The 4 wavelengths are diverse between an east and west path but they are not protected so it’s up to the higher layer protocols to provide the redundancy and failover.Not visible in the diagram above is a 10GE WAN ring that connects all our hospitals together. The primary and secondary data centers are also tied into that ring via multiple peering points for redundancy. You might be asking yourself why I’m using a Cisco 3750E as a termination switch in our primary data center. At the time we deployed our Cisco Nexus 7010s they didn’t support the 10GBase-ER SFP+ optic so I had to use the Cisco 3750E (with RPSU) as a glorified media transceiver/converter from 10GBase-ER to 10GBase-SR. The Cisco Nexus 7010 now has a 10GBase-ER SFP+ optic available so we didn’t need to use the Cisco 3750 in the secondary data center.

We are essentially stretching a Layer 2 vPC connection between the 2 data centers. It’s possible that some folks will get excited at the mention of Layer 2 between the data centers but it’s the best solution for us at this time and it certainly has pros and cons like everything in networking. We looked at potentially running OTV between the Cisco Nexus 7010s but ultimately decided to use a vPC configuration. We are only stretching the virtual machine VLANs that we need between the data centers.

My Thoughts

There’s a lot of work required to design any data center or even an ICR (Intermediate Communications Room), CCR (Central Communications Room), MDF (Main Distribution Frame) or IDF (Intermediate Distribution Frame). You’re immediately confronted with space, power and cooling challenges never mind coming up with the actual IP addressing scheme, VLAN assignments, routing vs bridging ,etc. You need to determine how much cabling you’ll need both CAT6 and fiber, perhaps you’ll look to use twinax of DAC (Direct Attach Copper) for your 10GE connections. Let’s not forget to include the ladder racks, basket trays, fiber conduits, PDUs, out-of-band networking, etc.

You also need to design the data center as if it was 300+ miles away… license those iLOs (HP Integrated Lights Out), purchase IP enabled KVMs, purchase console/terminal servers (Opengear or MRV) and wire everything up as if you will never have the opportunity to visit it again. We’ve had a few issues in the past few years that were quickly (less than 15 minutes) resolved thanks to having all our iLOs licensed, all our KVMs IP enabled, all our console/serial ports connected to a console/terminal server and the ability to dial-up into the console/terminal server should the problem get really bad.

Here’s a short story… We had a number of billing issues in the first few months of our contract with our current primary data center provider and the data from our Liebert PDUs, HP PDUs, and HP C7000 enclosures was invaluable in calling into question the numbers that were being reported to us. In all honesty when they told me we were consuming 53A on a 50A circuit I knew that something was grossly wrong with their math. In the end the provider admitted that there numbers were grossly wrong and the corrected numbers were in-line with the data we collected from our equipment.

It’s never a good idea to skimp on the documentation and I really advise taking lots of pictures, you’d be surprised how quickly you can forget what the back a specific rack looks like when you’re trying to walk Smart Hands through replacing a component at 2AM in the morning.

Cheers!

vSphere ESXi 4.1 U1 NIC Teaming Cisco Nexus vPC

by

I had a very interesting problem this past week that I thought I would share and even seek comment on. We have quite a few vSphere ESX 4.0 and even vSphere ESX 4.1 hosts running in our network. This past week we installed 2 HP DL-380 servers with vSphere ESXi 4.1 Update 1 and we immediately noticed an issue with NIC teaming on the management interface (vmk0). We were connecting the ESXi 4.1 hosts to a pair of Cisco Nexus 7010 switches over a virtual port-channel (vPC). I should comment that this is our first ESXi deployment, all previous VMware deployments had been ESX.

Loss of connectivity

In the past we’ve taken different approaches with respect to how to configure the core/distribution network switches when connecting them to ESX servers for NIC teaming. In some instances we’ve created SMLTs, MLTs (Nortel/Avaya) or vPCs, PCs (Cisco) depending on the location and equipment. In this instance no matter how we configured the switch port we would always loose network connectivity to the management interface the moment we brought up the second NIC. We had VMware ESX 4.1 hosts running from the exact same Cisco Nexus 7010s so I was at a loss to immediately explain the problem.

Solution – VMware KB Article

Thankfully I’ve become a master (as have many other technical folks) in the use of Google and pretty quickly stumbled across VMware KB article 1022751 entitled, NIC teaming using EtherChannel leads to intermittent network connectivity in ESXi.

When trying to team NICs using EtherChannel, the network connectivity is disrupted on an ESXi host. This issue occurs because NIC teaming properties do not propagate to the Management Network portgroup in ESXi. When you configure the ESXi host for NIC teaming by setting the Load Balancing to Route based on ip hash, this configuration is not propagated to Management Network portgroup.

I’ll be the first to admit that I haven’t spent too much time with ESXi, but seeing that VMware has already commented the ESX will be going way I probably need to do some catching up.

Cheers!

Cisco Nexus 7010 with BGP over vPC fails

by

I recently tried standing up a Cisco 3825 router attached to a Cisco 3750E switch which was in turn connected via vPC to a set of Nexus 7010 switches. I spent the better part of two days trying to get the BGP peers/neighbors to establish between the two Cisco Nexus 7010 switches and the Cisco 3825 router. It was really bizarre in that I was able to ping every interface involved so I had Layer 3 connectivity yet only one of the Nexus 7010 switches could establish a BGP neighbor with the 3825 router. The keepalive timer kept expiring on the second Nexus 7010 switch. After a few days I opened a case with Cisco and a week later I was informed that the configuration I was trying to implement was not supported (didn’t work).

Layer 3 and vPC Recommendations

I was provided a copy of the Nexus 7000 virtual Port-Channel Best Practices & Design Guidelines which clearly indicates on page 25 that routers should not be connected to a vPC link but should instead be connected via a Layer 3 switch port. Here are some bullet points;

  • Use separate L3 links to hook up routers to a vPC domain is still standing.
  • Don’t use L2 port channel to attach routers to a vPC domain unless you can statically route to HSRP address
  • If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2 port-channel for bridged traffic

I was still currious to understand more of the inner-workings.. why didn’t it work or wasn’t it allowed? I only had to flip through the next few slides although I can really say that I completely understand just yet.

  1. Packet arrives at R
  2. R does lookup in routing table and sees 2 equal paths going north (to 7k1 & 7k2)
  3. Assume it chooses 7k1 (ECMP decision)
  4. R now has rewrite information to which router it needs to go (router MAC 7k1 or 7k2)
  5. L2 lookup happens and outgoing interface is port-channel 1
  6. Hashing determines which port-channel member is chosen (say to 7k2)
  7. Packet is sent to 7k2
  8. 7k2 sees that it needs to send it over the peer-link to 7k1 based on MAC address
  9. 7k1 performs lookup and sees that it needs to send to S
  10. 7k1 performs check if the frame came over peer link & is going out on a vPC.
  11. Frame will only be forwarded if outgoing interface is NOT a vPC or if outgoing vPC doesn’t have active interface on other vPC peer (in our example 7k2)

I’m not embarrassed to say that I followed everything up until step 11. Why exactly is it that frames will only be forwarded if the outgoing interface is NOT a vPC or if the outgoing vPC doesn’t have an active interface on another vPC peer? Isthere anyone that can shed any additional light on this topic?

I’ve never experienced such a restriction in all my years of working with the Avaya (formerly Nortel) Ethernet Routing Switch 8600 and their Split Multilink Trunking (SMLT) technology. I actually have a Cisco 3825 router connected via a SMLT attached Ethernet Routing Switch 5520 (Layer 2) with the Cisco 3825 and the Avaya 8600s all running BGP.

Cheers!

New Data Center – Tidbits

by

I thought I would follow-up my last post with some additional tidbits and background on the data center build.

Let me start by saying that we haven’t abandoned Nortel/Avaya as a viable network electronics vendor. When we first started this effort back in January 2009 we initially looked at Brocade, Cisco, HP, Juniper and Nortel. After the initial cut we were left with Brocade, Cisco and Nortel/Avaya. When the final decision was made, back in November 2009, there was still a lot of concern about Avaya’s purchase of Nortel’s Enterprise Division so the decision really came down to Brocade or Cisco. Ultimately the decision was made to go with Cisco. As I commented in the previous post we still have well over 33,000 ports on Nortel/Avaya switches and less than 1,000 ports on Cisco switches. We are still purchasing and installing/deploying Nortel/Avaya switches in our hospitals and business offices .

Nexus 5010 Virtual PortChannel bug?

I’ve been known to tell people that we’re a partner with Nortel/Avaya because of the number of bugs that we identify and help to resolve.  Well if I thought it was going to be any different with Cisco I can throw that idea right out the window as we have identified a bug in the vPC code for the Nexus 5000 switch. We discovered the issue with a pair of Nexus 5010 switches when one of the switches up and died (hardware failure). While trying to troubleshoot the problem we restarted the remaining Nexus 5010 switch only to find that the remaining Nexus 5010 would no longer adopt the Nexus 2148 switches/FEXes after it was rebooted if the peer Nexus 5010 wasn’t online. If we removed the vpc configuration from the portchannel then the FEX would come online. After speaking with Cisco they have created bug CSCth01969 which is supposedly going to be addressed in the next software release.

HP Virtual Connect Flex-10/NIC issues/ESX 4.0 Update 1 Drivers

We also ran into a number of issues with the HP BL490c blades and HP Virtual Connect Flex-10 Interconnect. On a number of blades we had an issue where the ESX installer was unable to locate any network interfaces. The installer would error with something like “The script 32.networking-drivers failed to execute and the installation can not continue.” The solution was to re-apply the NIC firmware after the Virtual Connect profile had been applied to the server. You can find additional detail over at Brian’s blog if interested.

We also ran into another issue when performing our redundancy and high-availability testing. We configured Virtual Connect to use SmartLink, however, when we unplugged the uplinks from the B side interconnect the Virtual Connect Flex-10 interconnect didn’t disable the server side NICs. The solution to this problem was to update the NIC driver that shipped with ESX 4.0 Update 1. You can find the driver here.

Lots of additional stories to come including a comparison of vPC and SMLT.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL