Field Notice: FN – 72150 – Nexus 9000/3000 Will Fail With SSD Read-Only Filesystem – Power Cycle Required – BIOS/Firmware Upgrade Recommended

The issue was further compounded by some sloppy management, with several switches having unsaved configurations or having crashed and rebooted with unsaved configurations and ultimately inconsistent VPC states. In the short term I ended up deploying the SSD firmware update to all the impacted Cisco Nexus 9000 series switches in my network. I’ll look at performing the recommended software upgrades early next year.

You can setup notifications on the Cisco website to help keep you informed of field notices, software releases and security bulletins.

Anyone else run into this problem?

Cheers!

Upon reviewing the 9.1.7 release notes from Cisco I stumbled over the following entry;

CSCvd78303 – ARP functions fail after 213 days of uptime, drop with error ‘punt-rate-limit-exceeded’

We upgraded a large number of our Cisco ASAs about 213 days ago to 9.1.(7)12 and now it would seem we’ll be upgrading again to 9.1.(7)19 – the actual issue is resolved in 9.1(7)16.

Here’s the text of the bug report;

Symptom:
An ASA, after reaching an uptime of roughly 213 days will fail to process ARP packets leading to a condition where all traffic eventually stops passing through the affected device. Since not all existing ARP entries time out at the same time, not all connections may fail at the same time.
Additional symptoms include:

• ASA does not have ARP entries in its ARP table. show arp is empty
• The output of show asp drop and ASP drop captures indicate a rapidly increasing counter for punt-rate-limit exceeded and the dropped packets are predominantly ARP.

IMAGES WITH FIXES
Images with fixes for this defect will be published as soon as they are available, and posted to the Cisco Software Download center.
Conditions:
This is seen when the ASA’s uptime reaches 213 days.
This problem affects ASA and FTD versions:
ASA version 9.1 releases 9.1(7)8 and higher
ASA version 9.2 releases 9.2(4)15 and higher
ASA version 9.4 releases 9.4(3)5 and higher including 9.4(4)
ASA version 9.5 releases 9.5(3) and higher
ASA version 9.6 releases 9.6(2)1 and higher including 9.6(3)
ASA version 9.7 releases 9.7(1) and higher
FTD version 6.1 releases 6.1.0.1 and higher
FTD version 6.2 releases 6.2.0

Workaround:
Perform a pre-planned reboot of the device before approaching the 213 days (5124 hours) of up time. After the reboot, it will give you another 213 days of up time.
Further Problem Description:
Devices encountering this issue will not receive or respond to ARP packets. This affects not just transient traffic, but also access to the affected device – including Administration access such as SSH, HTTPS and Telnet. Console access is not affected.

If your running a Cisco ASA firewall I would recommend you check to make sure that you won’t be impacted by this bug.

Cheers!

The laptop would often connect to the SSID but the laptop would be unable to get a webpage to render with all IP traffic essentially stalling. ICMP ping times would jump from 1 ms to 3,900 ms with multiple dropped packets scattered all about the constant ping. Without any load you could occasionally get 1 ms response times for a couple of minutes at a time but the instant you opened a web page the traffic would stall and the ICMP pings would start timing out.

The Intel engineer that was assisting me provided the hint, letting me know that Cisco IT had actually stumbled across this very same issue the week earlier internally with their own employees. Cisco had intentionally disabled A-MPDU on their WLCs, the workaround was to enable A-MPDU for 802.11n on their WLCs. I went ahead and checked our WLCs and sure enough we also had A-MPDU disabled – not exactly sure who or why it was disabled.

802.11n Status: 
    A-MPDU Tx: 
        Priority 0............................... Disabled 
        Priority 1............................... Disabled 
        Priority 2............................... Disabled 
        Priority 3............................... Disabled 
        Priority 4............................... Disabled 
        Priority 5............................... Disabled 
        Priority 6............................... Disabled 
        Priority 7............................... Disabled 
        Aggregation scheduler.................... Enabled 
        Frame Burst.............................. Automatic 
            Realtime Timeout..................... 10 
    A-MSDU Tx: 
        Priority 0............................... Enabled 
        Priority 1............................... Enabled 
        Priority 2............................... Enabled 
        Priority 3............................... Enabled 
        Priority 4............................... Enabled 
        Priority 5............................... Enabled 
        Priority 6............................... Disabled 
        Priority 7............................... Disabled 
    Rifs Rx ..................................... Enabled 
    Guard Interval .............................. Any 

I used the following CLI commands to enable A-MPDU; (note that I had to temporarily disable the 802.11a network to make the change – you’ll want to schedule this off-hours)

config 802.11a disable 
y 
config 802.11a 11nsupport a-mpdu tx priority 0 enable 
config 802.11a 11nsupport a-mpdu tx priority 1 enable 
config 802.11a 11nsupport a-mpdu tx priority 2 enable 
config 802.11a 11nsupport a-mpdu tx priority 3 enable 
config 802.11a 11nsupport a-mpdu tx priority 4 enable 
config 802.11a 11nsupport a-mpdu tx priority 5 enable 
config 802.11a enable 

Why doesn’t the Intel AC 8260 wireless adapter negotiate using A-MSDU?

I hope to be able to bring you that answer from either Cisco or Intel.

I hope you enjoyed the article Tim.

Cheers!

Update: December 7, 2016

Intel has released a new driver for the AC 8260 that is designed to address the issue.
https://downloadcenter.intel.com/download/26465/Intel-PROSet-Wireless-Software-and-Drivers-for-Windows-10
https://downloadcenter.intel.com/download/26469/Intel-PROSet-Wireless-Software-and-Drivers-for-IT-Admins

I’m currently testing the driver but haven’t had enough time to comment yet.

If you are going to be adding new APs you had better make sure that you upgrade the software on your WLC first.

Cheers!

Update: Thursday September 8, 2016

It turned out we had to disconnect the APs for about 5 minutes to allow the DTLS cache on the controller to age out before the APs would join properly after upgrading the WLC.

On Wednesday morning I had two Cisco 5508 Wireless LAN Controllers both crash with a “Bonjour_Process_Task” taking too much cpu: 100% error message. It turns out that this is a known issue (CSCux78464 WLC crashes in Process Bonjour_Process_Task) that is resolved in 8.0.135.1, an engineering release which you need to contact Cisco TAC to obtain. If that wasn’t enough excitement for the morning I quickly noticed that of 120 APs that we usually have connected to the WLC we only had about 70 APs connected.  A quick examination of the debug logs (debug capwap errors enable) showed that multiple APs were failed to join the controller with messages like “Discarding non-ClientHello Handshake or DTLS encrypted packet” and “DTLS session is not established”. A quick call to Cisco TAC revealed that there are built-in certificates into the APs that can expire over time and that’s what had essentially happened. The certificates had expired since the APs had last joined the WLC and now that the certificates were expired they were not able to join the controller. Thankfully there’s a command in the CLI to ignore the certificate expiration;

config ap cert-expiry-ignore mic enable

With that command configured on the WLC the APs starting joining the controller and all was well again.

The field notice from Cisco providing all the details can be found here.

Cheers!

A year ago I was asked to investigate why VoIP calls were experiencing poor audio quality across an AT&T AVPN (Ethernet to MPLS) WAN link. After some research and examining the router configuration and logs I realized that the router had never been configured properly to support the 1Gbps access and 100Mbps port. While the router was physically connecting to the AT&T Cisco ONS via 1Gbps, AT&T was only allowing 100Mbps worth of traffic to pass through that connection. In this specific case traffic shaping wasn’t setup to properly limit the amount of traffic through the interface. I also found some occasions in the router logs where the BGP session was flapping due to BFD packets being lost between our router and AT&T’s router, again because QoS and traffic shaping hadn’t been setup properly on the Cisco router and the traffic was bursting well past the 100Mbps allocated from AT&T and that traffic was being dropped by AT&T.

The solution was to chain multiple policies together, first the traffic shaping policy and then the QoS policy. Here’s an example configuration I use on an AT&T AVPN 1Gbps Access with 100Mbps Port with 4 CoS queues assigned at 20% RT, 60/30/10. You need to shape the traffic to 100Mbps and then apply the QoS policies once you’ve shaped the traffic.

class-map match-any COS1
  description REAL-TIME VOICE
  match precedence 5
class-map match-any COS2
  description STREAMING VIDEO
  match precedence 4 6 7
class-map match-any COS3
  description CALL SIGNALLING
  match precedence 2 3
class-map match-any COS4
  description BEST EFFORT
  match precedence 0 1

policy-map QOSPOLICY
  description ATT Profile 113 20%RT/60/30/10
  class COS1
    priority 20000
  class COS2
    bandwidth remaining percent 60
  class COS3
    bandwidth remaining percent 30
  class class-default
    fair-queue

policy-map SHAPEPOLICY
  class class-default
    shape average 100000000
    service-policy QOSPOLICY

interface GigabitEthernet0/1
  bandwidth 100000
  ip address 10.x.x.x 255.255.255.252
  service-policy output SHAPEPOLICY

With this configuration in place the BFD keepalives stopped tripping false positives on the BGP session and VoIP packets were no longer getting starved for the bandwidth they needed.

Cheers!

This material has been covered hundreds if not thousands of times on the Internet by engineers and writers much more skilled than myself. However, I’m getting desperate for content here fellas and I’m running out of life stories.

I had explained to the client that unless they had multiple paths (routes) between their geographically located facilities they probably didn’t need a dynamic routing protocol. The client explained that every time they added a VLAN to any of their sites they had to go around to all their routers and update the static routes. I asked them if they had tried using summarized static routes. I explained to them that if they had laid out their IP address space appropriately they could utilize summary static routes as opposed to creating a route for every individual network or VLAN.

In the example above I have three physical locations. If I keep the IP addressing at each site within the Class B network then I can use a summary static route which will cover all future networks that I might want to route for in the future. A summary static route of 10.100.0.0/16 would cover any network that had started with 10.101.x.x. In the end it turned out that they had deployed their IP addressing in a rather haphazard fashion that wasn’t going to allow the use of summary static routes. In order to help their staff understand how dynamic routing worked I built a simple GNS3 lab with a simple hub and spoke topology simulating a single central site with two remote sites. I kept it super simple, and explained the difference between dynamic and static routing and how EIGRP built the routing table around it’s discovery of neighboring routers and networks.

The use of GNS3 allowed the staff members to interact with the test environment without fear of blowing something up in production and allowed them to build their confidence as we played around with various configurations, intentionally and occasionally accidentally breaking things to see their effect on routing.

In the end we only need a few commands to enable EIGRP;

router eigrp 1
passive-interface FastEthernet0/0
network 10.0.0.0
no auto-summary

With that configuration applied to all three routers we had dynamic routing up and running in our GNS3 lab configuration. When it came time for the actual deployment into the production network we ran into a small problem with their guest wireless network. They had used the same IP address space at each facility and it was routable via the local router so we had to exclude the IP network from participating in EIGRP by changing the inverse netmask of the network command. Here’s an example for the topology above;

network 10.100.0.0 0.0.255.255
network 10.150.0.0 0.0.255.255
network 10.200.0.0 0.0.255.255

We also added a few ACLs to help prevent the guest wireless network from accessing the internal corporate network.

With that tweak complete we had everything up and running and we proceeded to remove the static routes from all the configurations. If you’re interested you can download the sample GNS3 (v0.87) topology from this link, gns3-test.

I advise clients to think carefully how they layout their IP addressing schema. It can be painful and expense to change IP addresses after you’ve deployed your network.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 22 of 30. All the posts can be viewed from the 30in30 tag.

Last year as I was contemplating my next career move I went out and secured a number of basic certifications.

When I landed my current job my new employer didn’t too place much weight on industry certifications but instead relied on my reputation within the industry and my many years of experience and success in deploying and managing large and small IT environments. Since my new employer had a sizable deployment of Cisco networking equipment I thought it would be a good idea to continue my training and secure my CCNP – the knowledge would actually be useful. Now more than 11 months later I need to decide if I’m going to try and cram in all three exams or if I should wait until after the exams change. Or perhaps I should just abandon my plans to achieve the CCNP.

Thankfully I know most of the material fairly well although I’ll need to dedicate a sizable amount of time refreshing my knowledge of all the various topics and details. I’ll also have to brush up on my binary math and subnetting topics which always come up on these exams. I’m getting excited about the challenge just writing about it here but I also need to take into account my already busy and packed schedule. This might be a great opportunity of putting GNS3 1.2 to the test, or even perhaps trying out Cisco VIRL.

What to do is the question? Any ideas? I’m thinking of going for it, I actually have some vacation time coming up that I’ll be able to use.

Cheers!
Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 11 of 30. All the posts can be viewed from the 30in30 tag.

In this specific case all the routing was being performed by a number of high-end Cisco ASA firewalls which didn’t have PIM routing configured or enabled so I took the easy approach of just disabling IGMP snooping across the Cisco Catalyst 6504 and 4948 switches and the problem was solved. The cleaner solution would have been to setup an Mutlicast Router (mrouter) on the VLAN to properly handle all the IGMP requests and reports.

As pointed out by a colleague you can use a great little Python script written by RedHat for testing Multicast on your Linux servers.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 3 of 30. All the posts can be viewed from the 30in30 tag.

Reference;
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/68131-cat-multicast-prob.html
http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/route_multicast.html

Once we made the BGP local-pref change all IP traffic was egressing the AT&T circuit as visualized below.

There was never a problem reaching any of the ARIN IP address blocks that we were advertising via BGP the problem was isolated to just the WAN IP interfaces of the other Internet Service Providers.

The problem turned out to be that traffic to the WAN IP address was ingressing the circuit that the IP address was assigned to but it would egress the AT&T circuit due to the BGP local-pref statement. I’m guessing that AT&T is filtering the traffic on ingress checking for traffic sourced from an IP address block that has no business coming from that link and was dropping the traffic.

So an ICMP packet to the Comcast WAN IP address would ingress the Comcast interface and would egress the AT&T interface with a source IP address of the Comcast WAN interface. That packet would hit the AT&T head-end router which would discard any packets not sourced from the a valid ARIN IP address block belonging to that link, similar to Reverse Path Forwarding. I was able to verify this by placing a pair of static routes on the router using the Comcast circuit as a return path and with that the two Linux servers were now able to ping all the WAN IP interfaces. I’m guessing that while AT&T does some source route filtering, Comcast isn’t doing any.

It think it’s great that AT&T is filtering their inbound traffic for valid source IP blocks, it definitely helps prevent IP spoofing.

The confusion came when I did a debug ip icmp and later a debug ip packet 100 detail and observed no ICMP traffic coming from either of the two Linux servers on the Cisco ASR1001. I had a ticket open with Cisco TAC and they were also unable to explain the oddity. I’m curious if this was something to-do with CEF and might I need to enable no ip route-cache on the specific interfaces?

Cheers!

We opened a case with Cisco TAC and discovered that we were experiencing a known issue;

• CSCuj17283 – Cisco AP3700 used 8 replay counters with clients that support only one (ARP failed).

The bug above is resolved in software release 7.6.110, here are the release notes. We’re currently testing software release 7.6.110 although Cisco has just released 7.6.120 – release notes.

If your going to be deploying the Cisco 3700 Access Points you’ll want to make sure that your WLC is running a minimum software release of 7.6.110.

Cheers!

I decided to write about this topic since I recently encountered operational “difficulties” with both my prior employer and my current employer that involved the same near identical mistake of a network engineer accidentally overwriting the list of allowed VLANs. In the most recent case it was a simple oversight on the engineer’s part and the problem was quickly corrected. On the prior case the engineer had issued the command “no switchport trunk allowed vlan x” which seems to have given NXOS a bit of a fit. The ports that were in that vPC needed to be shutdown and then enabled to clear what appeared to be a software bug. While the running-config indicated that the VLANs were being trunked on the ports, the MAC/FDB table had no entries of those VLANs on the affected ports.

I strongly recommend that folks prune VLANs that aren’t being used from their trunks, however, you need to be very careful with how you add and/or remove VLANs from the list once the trunk is up and running.

In the past I’ve seen folks accidentally overwrite the VLAN allowed add list by using the “switchport trunk allowed vlan” command. Look at this sample configuration;

interface port-channel2
  description VPC_CISCO_NEXUS_5010
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 150-154
  switchport trunk allowed vlan add 155
  mtu 9216
  vpc 2

Now let’s say we wanted to add VLAN 156 and we using the following command, taking the allowed VLAN list and adding VLAN 156;

switchport trunk allowed vlan 152-154,156

The problem with this is that we just missed the fact that VLAN 155 was also on that trunk and we just removed it from the trunk with the that previous command.

The morale of the story – be careful when you add/remove VLANs from trunk ports and make sure you use the switchport trunk allow vlan add|remove command.

Cheers!

Image Credit: Roger Kirby

Before the BGP Soft Reset Enhancement feature, a soft reset for inbound routing table updates was performed by entering the neighbor soft-reconfiguration router configuration command. This command was used to configure the local BGP router to store all received (inbound) routing policy updates. However, this method uses too much memory because inbound updates are not modified and is not recommended.

I’m guessing this new feature had a significant impact for anyone taking a full Internet BGP table?

Cheers!

Image Credit: Earth 3D by Jan

Switch interfaces exchange PAgP packets only with partner interfaces configured in the auto or desirable modes. Switch interfaces exchange LACP packets only with partner interfaces configured in the active or passive modes. Interfaces configured in the on mode do not exchange PAgP or LACP packets.
…
Both the auto and desirable PAgP modes allow interfaces to negotiate with partner interfaces to determine if they can form an EtherChannel based on criteria such as interface speed and, for Layer 2 EtherChannels, trunking state and VLAN numbers.
…
Both the active and passive LACP modes allow interfaces to negotiate with partner interfaces to determine if they can form an EtherChannel based on criteria such as interface speed and, for Layer 2 EtherChannels, trunking state, and VLAN numbers.

We’ll be configuring our EtherChannel for LACP so we’ll use channel-group x mode active on both the Cisco 3750 and 2950 switches.

Sample Topology

Cisco Catalyst 3750E Switch

enable
config t
interface gig1/0/37
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active
interface gig1/0/38
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active

interface gig1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 4 mode active
interface gig1/0/48
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 4 mode active
exit
exit

Cisco Catalyst 2950 Switch 1 & 2

enable
config t
vlan 100
name "192-168-100-0/24"
exit
vlan 200
name "192-168-200-0/24"
exit

interface vlan 100
ip address 192.168.100.40 255.255.255.0
no shut
exit

spanning-tree mode mst

spanning-tree mst configuration
 name AcmeNetworks
 revision 1
 instance 1 vlan 100
 instance 2 vlan 200

interface fas0/1
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface fas0/2
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface port-channel 1
switchport mode trunk

interface fas0/31
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

interface fas0/32
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

interface port-channel 2
switchport mode trunk
exit
exit

That’s all well and good but I’m sure you want to see the output… is it working as expected?

Cisco Catalyst 2950 Switch #1

C2950-SW1#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0064.xxxx.4d80   7s    0x3     0x126    0x3D
Fa0/2     SA      32768     0064.xxxx.4d80  21s    0x3     0x127    0x3D

Channel group 2 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/31    SA      32768     0019.xxxx.49c0  23s    0x2     0x1F     0x3D
Fa0/32    SA      32768     0019.xxxx.49c0  21s    0x2     0x20     0x3D

C2950-SW1#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    16384
             Address     3475.xxxx.a400
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0018.xxxx.4a40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Desg FWD 100000    128.66   P2p

MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    16385
             Address     54e0.xxxx.d441
             Cost        110000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0018.xxxx.4a40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Desg FWD 100000    128.66   P2p

MST02
  Spanning tree enabled protocol mstp
  Root ID    Priority    16386
             Address     0064.xxxx.4d80
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0018.xxxx.4a40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Desg FWD 100000    128.66   P2p

Let’s have a look at the other Cisco 2950 switch;

Cisco Catalyst 2950 Switch #2

C2950-SW2#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0064.xxxx.4d80   9s    0x4     0x130    0x3D
Fa0/2     SA      32768     0064.xxxx.4d80  10s    0x4     0x131    0x3D

Channel group 2 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/31    SA      32768     0018.xxxx.4a40   4s    0x2     0x1F     0x3D
Fa0/32    SA      32768     0018.xxxx.4a40  25s    0x2     0x20     0x3D

C2950-SW2#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    16384
             Address     3475.xxxx.a400
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0019.xxxx.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p

MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    16385
             Address     54e0.xxxx.d441
             Cost        110000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0019.xxxx.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p

MST02
  Spanning tree enabled protocol mstp
  Root ID    Priority    16386
             Address     0064.xxxx.4d80
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0019.xxxx.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p

We can see that ports Fas0/31 and Fas0/32 that make up Port-Channel 2 are in Alternate Blocking mode on SW2. This is expected since the bridge address of SW1 (0018.xxxx.4a40) is lower than SW2 (0019.xxxx.49c0) and the links are equal cost.

Cheers!
Image credit: Ravenel Bridge Charleston by Roger Kirby

Let’s look at expanding the topology from our last post adding a Cisco 3750E;

Again, that’s pretty straight forward and isn’t too exciting. Although if we leave every uplink/downlink as a member of VLAN 100 and VLAN 200 we’ll end up with a loop in our topology – not a Spanning Tree Loop. What if we add Multiple Spanning Tree Protocol (MSTP) to our configuration just to make it interesting? Our topology might look like this with 2 instances of MSTP running, one for each VLAN.

We’ll make the Avaya switch the root bridge for CIST. We’ll make the Juniper switch the root bridge for MST 1, and we’ll make the Cisco switch the root bridge for MST 2.

That’s interesting… let’s see what we need to-do in order to configure everything up. I’m going to pickup the configuration as I had it setup in the previous post, LACP Configuration  Examples (Part 4). We’ll need to add another LACP group/pair to our Avaya and Juniper switches as well as configure the Cisco switch. We’ll also need to enable MSTP on each switch, add the VLANs to the correct MSTP instances and set the correct bridge priority for each.

Juniper EX2200-C Switch

configure
set chassis aggregated-devices ethernet device-count 2

delete interfaces ge-0/0/4 unit 0
delete interfaces ge-0/0/5 unit 0

set interfaces ge-0/0/4 ether-options 802.3ad ae1
set interfaces ge-0/0/5 ether-options 802.3ad ae1
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp periodic fast

set interfaces ae1 unit 0 family ethernet-switching
set interfaces ae1 unit 0 family ethernet-switching port-mode trunk
set interfaces ae1 unit 0 family ethernet-switching port-mode trunk vlan members VLAN-100 members VLAN-200

delete protocols rstp

set protocols mstp configuration-name AcmeNetworks
set protocols mstp revision-level 1
set protocols mstp msti 1 vlan 100
set protocols mstp msti 2 vlan 200

set protocols mstp msti 1 bridge-priority 16384
commit and-quit

Avaya Ethernet Routing Switch 5520

config t
spanning-tree mode mst
exit
boot

You’ll need to reboot the switch in order to enable MSTP, so go ahead and reboot before continuing the steps;

config t
vlan ports 25,26 tagging tagAll

interface fastEthernet 25,26
lacp key 25
lacp mode active
lacp timeout-time short
lacp aggregation enable
exit

spanning-tree mstp msti 1
spanning-tree mstp msti 1 add-vlan 100
spanning-tree mstp msti 2
spanning-tree mstp msti 2 add-vlan 200
spanning-tree mstp priority 4000

You’ll notice that the Avaya switch accepts a hexadecimal value for the priority, so 4000 in hex = 16384 in decimal.

spanning-tree mstp region region-name AcmeNetworks
spanning-tree mstp region region-version 1
exit

Cisco Catalyst 3750E Switch

config t
vlan 100
name "192-168-100-0/24"
exit
vlan 200
name "192-168-200-0/24"
exit

interface vlan 100
ip address 192.168.100.30 255.255.255.0
no shut
exit

interface vlan 200
ip address 192.168.200.30 255.255.255.0
no shut
exit

interface gig1/0/13
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface gig1/0/14
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface gig1/0/25
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

interface gig1/0/26
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

spanning-tree mode mst

spanning-tree mst configuration
name AcmeNetworks
revision 1
instance 1 vlan 100
instance 2 vlan 200
exit
spanning-tree mst 2 priority 16384
exit

Let’s have a look at our work and see what everything looks like from both a LACP and Spanning Tree perspective.

Cisco Catalyst 3750E Switch

Switch#show lacp neighbor
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/13 FA 127 54e0.xxxx.d440 5s 0x0 0x2 0x3 0x3F
Gi1/0/14 FA 127 54e0.xxxx.d440 5s 0x0 0x2 0x4 0x3F

Channel group 2 neighbors

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/25 FA 32768 3475.xxxx.a400 14s 0x0 0x3019 0x19 0x3F
Gi1/0/26 FA 32768 3475.xxxx.a400 16s 0x0 0x3019 0x1A 0x3F

Switch#show spanning-tree

MST0
Spanning tree enabled protocol mstp
Root ID Priority 16384
Address 3475.xxxx.a400
Cost 0
Port 496 (Port-channel2)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address 0064.xxxx.4d80
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 10000 128.488 P2p
Po2 Root FWD 10000 128.496 P2p

MST1
Spanning tree enabled protocol mstp
Root ID Priority 16385
Address 54e0.322a.d441
Cost 10000
Port 488 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 0064.xxxx.4d80
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 10000 128.488 P2p
Po2 Desg FWD 10000 128.496 P2p

MST2
Spanning tree enabled protocol mstp
Root ID Priority 16386
Address 0064.xxxx.4d80
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 16386 (priority 16384 sys-id-ext 2)
Address 0064.xxxx.4d80
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 10000 128.488 P2p
Po2 Desg FWD 10000 128.496 P2p

We can see that LACP is up and running to both the Avaya and Juniper switches. We can also see that the Cisco switch is the root bridge for MSTI 2 and the root port for MSTI 1 is Port-channel 1 (link to Juniper EX2200-C) while the root port for the CIST is Port-channel2 (link to Avaya ERS 5520). All ports are designated and forwarding traffic.

 Juniper EX2200-C Switch

root> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Fast Active
ge-0/0/1 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/1 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Fast periodic Collecting distributing
ge-0/0/1 Current Fast periodic Collecting distributing

Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/4 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/4 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/5 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/5 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/4 Current Slow periodic Collecting distributing
ge-0/0/5 Current Slow periodic Collecting distributing

root> show spanning-tree bridge

STP bridge parameters
Context ID : 0
Enabled protocol : MSTP

STP bridge parameters for CIST
Root ID : 16384.34:75:xx:xx:a4:00
Root cost : 0
Root port : ae0.0
CIST regional root : 16384.34:75:xx:xx:a4:00
CIST internal root cost : 10000
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Hop count : 19
Message age : 0
Number of topology changes : 2
Time since last topology change : 14690 seconds
Topology change initiator : ae0.0
Topology change last recvd. from : 34:75:xx:xx:a4:01
Local parameters
Bridge ID : 32768.54:e0:xx:xx:d4:41
Extended system ID : 0
Internal instance ID : 0

STP bridge parameters for MSTI 1
MSTI regional root : 16385.54:e0:xx:xx:d4:41
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Number of topology changes : 5
Topology change initiator : ae1.0
Topology change last recvd. from : 00:64:xx:xx:4d:8d
Local parameters
Bridge ID : 16385.54:e0:xx:xx:d4:41
Extended system ID : 0
Internal instance ID : 1

STP bridge parameters for MSTI 2
MSTI regional root : 16386.00:64:xx:xx:4d:80
Root cost : 10000
Root port : ae1.0
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Hop count : 19
Number of topology changes : 6
Topology change initiator : ae1.0
Topology change last recvd. from : 00:64:xx:xx:4d:8d
Local parameters
Bridge ID : 32770.54:e0:xx:xx:d4:41
Extended system ID : 0
Internal instance ID : 2

Avaya Ethernet Routing Switch 5520

5520-48T-PWR#show lacp port 13-14,25-26
Admin Oper Trunk Partner
Port Priority Lacp A/I Timeout Key Key AggrId Id Port Status
---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
13 32768 Active A Short 1 12289 8224 32 1 Active
14 32768 Active A Short 1 12289 8224 32 2 Active
25 32768 Active A Short 25 12313 8223 31 282 Active
26 32768 Active A Short 25 12313 8223 31 283 Active

5520-48T-PWR#show spanning-tree mstp config
Maximum Mst Instance Number: 8
Number of Msti Supported: 2
Cist Bridge Priority (hex): 4000
Stp Version: Mstp Mode
Cist Bridge Max Age: 20 seconds
Cist Bridge Forward Delay: 15 seconds
Tx Hold Count: 3
Path Cost Default Type: 32-bit
Max Hop Count: 2000

VLAN members
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
1

Msti Config Id Selector: 0
Msti Region Name: AcmeNetworks
Msti Region Version: 1
Msti Config Digest: 6D:A4:B5:0C:4F:D5:87:75:7E:EF:03:56:75:36:05:E1

5520-48T-PWR#show spanning-tree mstp msti config 1
Msti Bridge Regional Root:  40:00:54:E0:xx:xx:D4:41
Msti Bridge Priority (hex): F000
Msti Root Cost:             10000
Msti Root Port:             MLT 32
Msti State:                 Enabled

VLAN members
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
100

5520-48T-PWR#show spanning-tree mstp msti config 2
Msti Bridge Regional Root:  40:00:00:64:xx:xx:4D:80
Msti Bridge Priority (hex): F000
Msti Root Cost:             10000
Msti Root Port:             MLT 31
Msti State:                 Enabled

VLAN members
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
200

5520-48T-PWR#show spanning-tree mstp msti port role 1
Port Role State STP Status Oper Status
---- ---------- ---------- ---------- -----------
13 Root Forwarding Enabled Enabled
14 Root Forwarding Enabled Enabled
25 Alternate Discarding  Enabled Enabled
26 Alternate Discarding  Enabled Enabled

5520-48T-PWR#show spanning-tree mstp msti port role 2
Port Role State STP Status Oper Status
---- ---------- ---------- ---------- -----------
13 Alternate Discarding  Enabled Enabled
14 Alternate Discarding  Enabled Enabled
25 Root Forwarding Enabled Enabled
26 Root Forwarding Enabled Enabled

We can see from the output above that ports 13,14 are Alternate Discarding for MSTI 1 while ports 25,26 are Alternate Discarding for MSTI 2.

In the output we can see which port is the root bridge port for each switch, we can also see the MSTP config digest which should match on every switch in the topology. In order for the configuration to be valid the MST region name, version and config selector need to match along with correct VLAN IDs matched to the correct MST instance.

Cheers!
Image Credit: New York City Brooklyn Bridge by Diogo Ferrari

My personal experience with Cisco is fairly new. I deployed a new data center almost two years ago based on dual Cisco Nexus 7010s with Nexus 5010s, Nexus 2148s, Catalyst 3750E, and Catalyst 2960G for 10/100Mbps. We deployed Cisco Catalyst 3120Xs in our non-VM HP C7000 enclosures and in our VMware vSphere ESX 4.1 environment we relied on the Cisco Nexus 1000V. We deployed redundant Cisco ASA 5520s to firewall our managed services provider and a pair of Cisco ACE 4710s to provide basic Layer 7 content load-balancing. There were a few issues with the 7010s, 5010s and 1000V but we were able to work through them and have been very stable for the past 18 months. In the past two months I’ve deployed a secondary data center duplicating the design of the first. I’m in the process putting that data center into production and hope to post more about it in the near future.

I’m going to outline the different presentations that we heard and perhaps make a few points here and there if I have anything useful to say. I’ll include a short blurb from Cisco in italics to help define/describe each product or solution. Thankfully since the sessions were recorded you can watch for yourself and make your own informed opinion.

Here’s my disclaimer; I’m not endorsing any of the solutions presented below. I’m merely passing on the information along with a few comments of my own. If you have any personal opinions about the solutions below why not share them with us in the comments?

Cisco UCS E-Series

Tech Field Day Video
by Anurag Gurtu

Cisco UCS E-Series Server modules are next-generation, power-optimized, x86, Intel® Xeon® 64-bit blade servers designed to be deployed in Cisco Integrated Services Routers Generation 2 (ISR G2). These price-to-performance-optimized, single-socket blade servers balance simplicity, performance, reliability, and power efficiency. They are well suited for applications and infrastructure services typically deployed in small offices and branch offices.

Cisco UCS E-Series Server modules are available in two form factors. The first is a single-wide blade server powered by a high-performance yet power-efficient Intel® Xeon® processor E3 with four cores, up to 16 Gigabytes of RAM, and two Terabytes of local storage. The second is a double-wide blade server powered by a high-performance Intel® Xeon® processor E5-2400 with up to six cores and support for up to 48 Gigabytes of RAM and three Terabytes of local storage.

My Thoughts?

In short it’s a server in a router and is compatible in the ISR G2 series (model 2911 and upward). It comes in single wide and double wide variants and is targeted at branch offices. It supports Microsoft Hyper-V, VMware vSphere, Citrix Xen Server and is available in bundled SKUs to help offset pricing. The license that is included in the bundled SKUs does not allow for Vmotion or VMware HA since it’s not expected that you would run a VMware cluster on this solution. I can definitely see this product filling a need in many organizations. There was some discussion among the delegates concerning the lack of VMware HA and Vmotion with the bundled VMware license. I’m not sure I see folks using this hardware in an HA configuration.

You can see a demo of the Cisco UCS-E in this Tech Field Day Video.

Cisco CSR 1000v (Cloud Services Router)

Tech Field Day Video
by Bopaiah PuliyvandaThe Cisco Cloud Services Router (CSR) 1000V Series is comprised of single-tenant software routers in virtual form factor that deliver comprehensive WAN gateway functionality to multi-tenant, provider-hosted clouds. With the Cisco CSR 1000V Series, enterprises can transparently extend a WAN to off-premises clouds, and cloud providers can offer enterprise-class networking services to their tenants.

My Thoughts?

It’s a software based router that runs on top of VMware ESXi or Citrix XenServer. It can be deployed as a secure VPN gateway, MPLS WAN termination point, DC network extension, or a control point for networking services (like WAN Optimization) for your hosted cloud. There have been some to say that it’s really a solution intended to provide an all-Cisco environment from campus edge to cloud.

The low-latency space has really been heating up over the past few years. I believe the Cisco Nexus 3548 and specifically the Monticello ASIC has been in development for almost 2 years and was released at the same time as Arista’s 7150S. Unfortunately I don’t see myself running any of these anytime soon… I had to fight to get 10Gbps in my Data Centers and I usually have to convince customers to go with 1Gps to the desktop as opposed to 100Mbps.

Shaun had the opportunity to test one of only a few test/evaluation units available and was kind enough to compile a video for all those interested.

You’ll find plenty of information from Cisco and even from the renowned Ivan Pepelnjak and Brad Reese.

Thanks Shaun!

I did notice a very odd problem while trying to copy the software to the switch via TFTP – it was insanely slow. It was copying the software at what appeared to be 8-16kbps. I issued a Ctrl-C and tried an FTP download and it flew along and I was done in minutes. In both cases I utilized the default VRF. I’m curious to understand why the TFTP was so slow compared to the FTP. We utilize TFTP pretty heavily in our environment and we’ve never had a problem with any other equipment so I suspect the Cisco Nexus 7010s and not the CentOS Linux server that acts as our central TFTP server.

The next big hurdle will be finding the downtime to apply all the EPLD/FPGA firmware upgrades for each card. I understand the EPLD upgrade is disruptive but I’m trying to determine how big a maintenance window I need in order to safely accomplish the task on both core Cisco 7010s – doing them one at a time. One Cisco resource I talked with said I would need a minimum 4 hour maintenance window per chassis – there’s no way in hell I’m going to get a four hour maintenance window in a healthcare environment.

Here are the commands and output in case anyone is curious or would like to compare notes.

sw-n7010-ccr.acme.org# install all kickstart bootflash:n7000-s1-kickstart. 5.2.4.bin system bootflash:n7000-s1-dk9.5.2.4.bin

Verifying image bootflash:/n7000-s1-kickstart.5.2.4.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n7000-s1-dk9.5.2.4.bin for boot variable "system".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n7000-s1-kickstart.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "cmp" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Extracting "cmp-bios" version from image bootflash:/n7000-s1-dk9.5.2.4.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS

Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes  non-disruptive       rolling
     2       yes  non-disruptive       rolling
     3       yes  non-disruptive       rolling
     4       yes  non-disruptive       rolling
     5       yes  non-disruptive         reset
     6       yes  non-disruptive         reset
     7       yes  non-disruptive       rolling  

Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1      lc1n7k                                    4.2(6)                5.2(4)           yes
     1        bios     v1.10.6(11/04/08):  v1.10.6(11/04/08)                                  no
     2      lc1n7k                                    4.2(6)                5.2(4)           yes
     2        bios     v1.10.6(11/04/08):  v1.10.6(11/04/08)                                  no
     3      lc1n7k                                    4.2(6)                5.2(4)           yes
     3        bios     v1.10.6(11/04/08):  v1.10.6(11/04/08)                                  no
     4      lc1n7k                                    4.2(6)                5.2(4)           yes
     4        bios     v1.10.6(11/04/08):  v1.10.6(11/04/08)                                  no
     5      system                                    4.2(6)                5.2(4)           yes
     5   kickstart                                    4.2(6)                5.2(4)           yes
     5        bios     v3.19.0(03/31/09):  v3.19.0(03/31/09)                                  no
     5         cmp                                    4.2(1)                5.2(4)           yes
     5    cmp-bios                                  02.01.05              02.01.05            no
     6      system                                    4.2(6)                5.2(4)           yes
     6   kickstart                                    4.2(6)                5.2(4)           yes
     6        bios     v3.19.0(03/31/09):  v3.19.0(03/31/09)                                  no
     6         cmp                                    4.2(1)                5.2(4)           yes
     6    cmp-bios                                  02.01.05              02.01.05            no
     7      lc1n7k                                    4.2(6)                5.2(4)           yes
     7        bios     v1.10.6(11/04/08):  v1.10.6(11/04/08)                                  no

Additional info for this installation:
--------------------------------------

Do you want to continue with the installation (y/n)?  [n] y

Install is in progress, please wait.

Syncing image bootflash:/n7000-s1-kickstart.5.2.4.bin to standby.
[####################] 100% -- SUCCESS

Syncing image bootflash:/n7000-s1-dk9.5.2.4.bin to standby.
[####################] 100% -- SUCCESS

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.
[####################] 100% -- SUCCESS

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 2: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 3: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 4: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 5: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 6: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 7: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS
2012 Mar 21 04:52:52 sw-n7010-ccr %$ VDC-1 %$ %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number JAXXXXXXXXX)
2012 Mar 21 04:58:30 sw-n7010-ccr %$ VDC-1 %$ %CMPPROXY-STANDBY-2-LOG_CMP_UP: Connectivity Management processor(on module 5) is now UP

Module 5: Waiting for module online.
 -- SUCCESS
2012 Mar 21 04:59:42 sw-n7010-ccr %$ VDC-1 %$ %IDEHSD-STANDBY-2-MOUNT: logflash: online 

Notifying services about the switchover.
[####################] 100% -- SUCCESS

"Switching over onto standby".
 writing reset reason 7, SAP(93): Swover due to install

NX7 SUP Ver 3.19.0
Serial Port Parameters from CMOS
PMCON_1: 0x200
PMCON_2: 0x0
PMCON_3: 0x3a
PM1_STS: 0x101
Performing Memory Detection and Testing
Testing 1 DRAM Patterns
Total mem found : 4096 MB
Memory test complete.
NumCpus = 2.
Status 61: PCI DEVICES Enumeration Started
Status 62: PCI DEVICES Enumeration Ended
Status 9F: Dispatching Drivers
Status 9E: IOFPGA Found
Status 9A: Booting From Primary ROM
Status 98: Found Cisco IDE
Status 98: Found Cisco IDE
Status 90: Loading Boot Loader
                                                                                                                                                                 Reset Reason Registers: 0x1 0x0
 Filesystem type is ext2fs, partition type 0x83

              GNU GRUB  version 0.97 

Autobooting bootflash:/n7000-s1-kickstart.5.2.4.bin bootflash:/n7000-s1-dk9.5.2.4.bin...
 Filesystem type is ext2fs, partition type 0x83
Booting kickstart image: bootflash:/n7000-s1-kickstart.5.2.4.bin.......................
............................................................................Image verification OK

ÿ
INIT: version 2
Checking all filesystems..r.r.r.. done.
Loading system software
/bootflash//n7000-s1-dk9.5.2.4.bin read done
Uncompressing system image: bootflash:/n7000-s1-dk9.5.2.4.bin Wed Mar 21 05:03:26 EDT 2012
blogger: nothing to do.

..done Wed Mar 21 05:03:30 EDT 2012
Load plugins that defined in image conf: /isan/plugin_img/img.conf
Loading plugin 0: core_plugin...
num srgs 1
0: swid-core-supdc3, swid-core-supdc3
num srgs 1
0: swid-supdc3-ks, swid-supdc3-ks

INIT: Entering runlevel: 3

User Access Verification
SW-N7010-CCR(standby) login:

Now we moved over to the standby supervisor which was slot 5 at the time to obverse the upgrade complete;

Continuing with installation, please wait

2012 Mar 21 04:58:30 sw-n7010-ccr %$ VDC-1 %$ %CMPPROXY-2-LOG_CMP_UP: Connectivity Management processor(on module 5) is now UP

Module 5: Waiting for module online.
 -- SUCCESS

2012 Mar 21 04:59:42 sw-n7010-ccr %$ VDC-1 %$ %IDEHSD-2-MOUNT: logflash: online 

2012 Mar 21 04:59:47 sw-n7010-ccr %$ VDC-1 %$ Mar 21 04:59:47 %KERN-2-SYSTEM_MSG: Switchover started by redundancy driver - kernel

2012 Mar 21 04:59:47 sw-n7010-ccr %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase).

2012 Mar 21 04:59:47 sw-n7010-ccr %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_START: Supervisor 5 is becoming active.

2012 Mar 21 04:59:49 sw-n7010-ccr %$ VDC-1 %$ %SYSMGR-2-SWITCHOVER_OVER: Switchover completed.

2012 Mar 21 05:00:00 sw-n7010-ccr %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Load time of /isan/etc/routing-sw/cli/metro.cli_: 695539ms - ascii_cfg_server

2012 Mar 21 05:04:52 sw-n7010-ccr %$ VDC-1 %$ %CMPPROXY-STANDBY-2-LOG_CMP_UP: Connectivity Management processor(on module 6) is now UP

2012 Mar 21 05:06:00 sw-n7010-ccr-b %$ VDC-1 %$ %IDEHSD-STANDBY-2-MOUNT: logflash: online 

Module 1: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 2: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 3: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 4: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 7: Non-disruptive upgrading.
[####################] 100% -- SUCCESS

Module 5: Upgrading CMP image.
Warning: please do not reload or power cycle CMP module at this time.
[####################] 100% -- SUCCESS

Module 6: Upgrading CMP image.
Warning: please do not reload or power cycle CMP module at this time.
[####################] 100% -- SUCCESS

Recommended action::
"Please reload CMP(s) manually to have it run in the newer version.".

Install has been successful.

With the upgrade complete the only thing we needed to-do was to restart the CMPs on slots 5 and 6;

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2012, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

sw-n7010-ccr.acme.org# reload cmp module 5
This command will reload the CMP on the supervisor in slot 5.  Continue (y/n)?  [no] y

sw-n7010-ccr.acme.org# 2012 Mar 21 05:23:41 sw-n7010-ccr-b %$ VDC-1 %$ %CMPPROXY-2-LOG_CMP_WENT_DOWN: Connectivity Management processor (on module 5) went DOWN
2012 Mar 21 05:24:50 sw-n7010-ccr %$ VDC-1 %$ %CMPPROXY-2-LOG_CMP_UP: Connectivity Management processor(on module 5) is now UP

sw-n7010-ccr.acme.org# reload cmp module 6
This command will reload the CMP on the supervisor in slot 6.  Continue (y/n)?  [no] y

sw-n7010-ccr.acme.org# 2012 Mar 21 05:25:50 sw-n7010-ccr-b %$ VDC-1 %$ %CMPPROXY-STANDBY-2-LOG_CMP_WENT_DOWN: Connectivity Management processor (on module 6) went DOWN
2012 Mar 21 05:26:57 sw-n7010-ccr %$ VDC-1 %$ %CMPPROXY-STANDBY-2-LOG_CMP_UP: Connectivity Management processor(on module 6) is now UP

Cheers!

Pre-Upgrade Script

The Pre-Upgrade Script, a TCL script which checks your Cisco Nexus 1000V configuration for any potential issues, immediately flag our Port Channel configurations in Check 2.

###############################################################################
## COMPANY NAME: Cisco Systems Inc                                           ##
## Copyright © 2011 Cisco Systems, Inc. All rights Reserved.                 ##
##                                                                           ##
## SCRIPT NAME: pre-upgrade-check-4.2(1)SV1(4).tcl                           ##
## VERSION: 1.0                                                              ##
## DESCRIPTION: This script is applicable to all releases prior to           ##
##              4.2(1)SV1(4).                                                ##
##                                                                           ##

...
...
...

=========
 CHECK 2:
=========
Checking for Interface override configurations on Port-channnel Interface ...
###############################################################################
##                           FAIL                                            ##
##                                                                           ##
## List of Port-Channel Interface(s) with interface level configuration(s)   ##

1: port-channel1 has below overrides.
mtu 9000

2: port-channel2 has below overrides.
mtu 9000

3: port-channel3 has below overrides.
mtu 9000

4: port-channel4 has below overrides.
mtu 9000

...
...
...

While originally testing the Cisco Nexus 1000V prior to going into production some 463 days earlier we had played around with enabling Jumbo Frame support within the VM environment. We had set the MTU on the individual port-channel configurations to 9000. Now the pre-upgrade script was telling us that we needed to clean this up and remove any specific configuration from the port-channel and instead apply it to the port-profile configuration. I added the system mtu 9000 command to the port-profile but got a few surprises when I tried to remove the MTU command. The first surprise when I issued the “inter po1, no mtu 9000” was loosing connectivity to the VM guests on that host. I had to manually restart the VEM on the ESX host with the “vem restart” command from an CLI prompt. The second surprise was that “mtu 9000” in the configuration had been replaced by “mtu 1500”. That wasn’t going to work so I had to reach out to Cisco TAC who immediately recognized the issue and provided a workaround;

On the ESX host stop the VEM

vem stop

Then on the Cisco Nexus 1000V delete the port-channels and associated VEM (I’ll use the first server as an example assuming there are two VSMs installed)

config
no inter po1
no inter po2
no vem 3

On the ESX host start the VEM

vem start

And sure enough it worked just as promised… recreating the port-channels without the MTU commands. Obviously we had to put each ESX host into maintenance mode before we could just stop and start the VEM.

With that taken care of we started upgrading the VEMs using Update Manager. Unfortunately Update Manager only made it through about 6 ESX hosts before it got stuck. We had to manually install the update VIB on the remaining 12 ESX hosts ourselves. We utilized FileZilla to copy the VIB up to each server and the utilized PuTTY to SSH into each server and manually update the VEM;

[root@esx-srv1-dc1-pa ~]# esxupdate -b ./cross_cisco-vem-v130-4.2.1.1.4.0.0-2.0.1.vib update
Unpacking cross_cisco-vem-v130-esx_4.2.1.1.. ############################################################# [100%]

Installing cisco-vem-v130-esx                ############################################################# [100%]

Running [rpm -e cisco-vem-v120-esx]...
ok.
Running [/usr/sbin/vmkmod-install.sh]...
ok.

With all the ESX hosts upgrade we had to physically reboot the vCenter server to get the currently running VUM task to fail so we could complete the upgrade from the Cisco Nexus 1000V.

Next we launched the upgrade application and before long we had the standby VSM upgraded to 4.2(1)SV1(4). Here’s where we ran into another small scare. After the upgrade of the standby VSM the upgraded VEMs are supposed to re-register to the newly upgraded VSM. We waited about 5 minutes an none of the VEMs had discnonected from the primary VSM running 4.0(4)SV1(3) to the standby VSM that was now running 4.2(1)SV1(4). It was only approximately 15-20 minutes later (while searching Google for some hint) that the VEMs just up and started to connect to the newly upgraded standby VSM.

Cheers!

Needless to say I never made it to the lab. I did have a few issues getting GNS3 going but those could probably be attributed to PICNIC (problem in chair not in computer). I also had some difficulties getting GNS3 to save all the router configurations, again probably a shortcoming of not reading all the documentation.

I’m

really impressed… while it can’t emulate any of the switches it does a great job of emulating the basic router hardware. The image you use in GNS3 is the actual IOS image that you download from Cisco’s website.

I’ll definitely be adding GNS3 to my list of recommended tools!

Cheers!

You’ll notice in the example below that I’m using two RADIUS servers, actually two Microsoft Internet Authentication Servers running Windows 2003. I created a SSID (or WLAN) of “love” and bridged it to VLAN 802. I had to utilize bridge group 254 because the bridge groups only go from 1-255. I also only configured the WLAN on the 802.11b/g radio (Dott11Radio0) and not the 802.11a radio (Dott11Radio1). I also utilized a RADIUS secret of “radiuspass” in the example below.

aaa group server radius acme_eap
 server 10.1.4.21 auth-port 1812 acct-port 1813
 server 10.2.4.21 auth-port 1812 acct-port 1813

aaa authentication login acme_methods group acme_eap

dot11 ssid love
   vlan 802
   authentication open eap acme_methods
   authentication network-eap acme_methods
   authentication key-management wpa 

interface Dot11Radio0

ssid love

encryption vlan 802 mode ciphers aes-ccm

interface Dot11Radio0.802
 encapsulation dot1Q 802
 no ip route-cache
 bridge-group 254
 bridge-group 254 subscriber-loop-control
 bridge-group 254 block-unknown-source
 no bridge-group 254 source-learning
 no bridge-group 254 unicast-flooding
 bridge-group 254 spanning-disabled

interface FastEthernet0.802
 encapsulation dot1Q 802
 no ip route-cache
 bridge-group 254
 no bridge-group 254 source-learning
 bridge-group 254 spanning-disabled

interface Dot11Radio0

ssid love

encryption vlan 802 mode ciphers aes-ccm

radius-server host 10.1.4.21 auth-port 1812 acct-port 1813 key radiuspass
radius-server host 10.2.4.21 auth-port 1812 acct-port 1813 key radiuspass
radius-server deadtime 5

If you need to debug the AAA or RADIUS process here are the commands that can help provide additional detail from the Access Point. It should be noted that some of the commands below are software and version dependent and might throw you an error.

debug dot11 aaa manager keys
debug dot11 aaa authenticator state-machine
debug dot11 aaa dot1x state-machine
debug dot11 aaa authenticator process
debug dot11 aaa dot1x process
debug radius authentication 

terminal monitor

While this example won’t translate directly for @fryguy_pa it might help others trying to deploy 802.1x WPA2 Enterprise with AES encryption in an enterprise network.

Cheers!

In this post I thought I would expand the network topology of my previous two posts to include a Cisco Catalyst 3750-E. I’ll specifically cover how to trunk (bridge) multiple VLANs between a stackable Avaya Ethernet Routing Switch and the Cisco Catalyst 3750-E and how to configure multiple interfaces in a Link Aggregation Group (LAG) utilizing LACP similar to Avaya’s proprietary MLT feature.

Avaya Ethernet Routing Switch 4548

enable
config t

Let’s start by making ports 45 and 46 trunk ports which will utilize 802.1Q tagging;

vlan ports 45,46 tagging tagAll

Let’s add the VLANs we wish to bridge across the trunk ports;

vlan members add 1 45,46
vlan members add 100 45,46
vlan members add 200 45,46

Now we’ll enable LACP on ports 45 and 46 using the same LACP key which will automatically create the LAG;

interface fastEthernet 45
lacp key 10
lacp mode active
lacp timeout-time short
lacp aggregation enable
exit

interface fastEthernet 46
lacp key 10
lacp mode active
lacp timeout-time short
lacp aggregation enable
exit

Avaya Ethernet Routing Switch 4548 – Show Commands

4548GT-PWR#show lacp port 45,46
Admin Oper         Trunk Partner
Port Priority Lacp    A/I Timeout Key   Key   AggrId Id    Port    Status
---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
45   32768    Active  A   Short   10    12298 8224   32    302     Active
46   32768    Active  A   Short   10    12298 8224   32    303     Active

4548GT-PWR#show mac-address-table
Mac Address Table Aging Time: 300
Number of addresses: 26

   MAC Address    Vid  Source         MAC Address    Vid  Source
----------------- ---- -------     ----------------- ---- -------
00-02-B3-CB-77-A2    1 Port:19     00-04-61-9E-46-7E    1 Port:21
00-0C-29-64-33-F9    1 Port:19     00-0C-29-A5-CB-54    1 Port:19
00-0F-20-95-38-D5    1 Port:11     00-18-01-EA-F4-45    1 Port: 1
00-1C-11-6B-DC-6B    1 Port: 1     00-1C-11-6D-15-27    1 Port: 1
00-1C-11-6D-15-DC    1 Port: 1     00-1E-7E-7C-2C-00    1
00-1E-7E-7C-2C-40    1             00-1F-0A-CE-BC-01    1 Trunk:1
00-1F-0A-CE-BC-40    1 Trunk:1     00-1F-D0-D0-BE-2D    1 Port:17
00-23-EE-96-AA-21    1 Port: 1     00-24-B5-F6-94-02    1 Trunk:1
00-64-40-CF-4D-AD    1 Trunk:32    00-64-40-CF-4D-AE    1 Trunk:32
00-64-40-CF-4D-C0    1 Trunk:32    00-0A-E4-76-9C-C8    2 Port:44
00-24-DC-DF-0D-08    2 Port:43     00-A0-F8-5E-CE-BC    2 Port:39
00-1F-0A-CE-BC-41  100 Trunk:1     00-24-7F-99-84-70  100 Port:25
00-64-40-CF-4D-AD  100 Trunk:32    00-1E-CA-F3-1D-B4  200 Port:26
00-1F-0A-CE-BC-43  200 Trunk:1     00-64-40-CF-4D-AD  200 Trunk:32

4548GT-PWR#show mlt
Id Name             Members                Bpdu   Mode           Status  Type
-- ---------------- ---------------------- ------ -------------- ------- ------
1  MLT_to_ERS5520   47-48                  All    Basic          Enabled Trunk
2  Trunk #2         NONE                   All    Basic          Disabled
3  Trunk #3         NONE                   All    Basic          Disabled
4  Trunk #4         NONE                   All    Basic          Disabled
5  Trunk #5         NONE                   All    Basic          Disabled
6  Trunk #6         NONE                   All    Basic          Disabled
7  Trunk #7         NONE                   All    Basic          Disabled
8  Trunk #8         NONE                   All    Basic          Disabled
9  Trunk #9         NONE                   All    Basic          Disabled
10 Trunk #10        NONE                   All    Basic          Disabled
11 Trunk #11        NONE                   All    Basic          Disabled
12 Trunk #12        NONE                   All    Basic          Disabled
13 Trunk #13        NONE                   All    Basic          Disabled
14 Trunk #14        NONE                   All    Basic          Disabled
15 Trunk #15        NONE                   All    Basic          Disabled
16 Trunk #16        NONE                   All    Basic          Disabled
17 Trunk #17        NONE                   All    Basic          Disabled
18 Trunk #18        NONE                   All    Basic          Disabled
19 Trunk #19        NONE                   All    Basic          Disabled
20 Trunk #20        NONE                   All    Basic          Disabled
21 Trunk #21        NONE                   All    Basic          Disabled
22 Trunk #22        NONE                   All    Basic          Disabled
23 Trunk #23        NONE                   All    Basic          Disabled
24 Trunk #24        NONE                   All    Basic          Disabled
25 Trunk #25        NONE                   All    Basic          Disabled
26 Trunk #26        NONE                   All    Basic          Disabled
27 Trunk #27        NONE                   All    Basic          Disabled
28 Trunk #28        NONE                   All    Basic          Disabled
29 Trunk #29        NONE                   All    Basic          Disabled
30 Trunk #30        NONE                   All    Basic          Disabled
31 Trunk #31        NONE                   All    Basic          Disabled
32 Trunk #32        45-46                  Single DynLag/Basic   Enabled Trunk

You might be looking at the output above and asking yourself what’s “Trunk 32”? Let me provide some quick background. You can have a total of 32 MLT/LAG trunks on a stackable Avaya Ethernet Routing Switch. When you create LACP trunks the switch automatically creates a LAG in the MLT table dynamically from the bottom up. While in the previous post I created “Trunk 1” by trunking ports 47 and 48 together (see above), in this post I’ve created an LACP trunk on ports 45 and 46 which will be reported it the switch as “Trunk 32”. You can also see it in the MAC/FDB table above.

Cisco Catalyst 3750-E

enable
config t

Let’s give the switch an IP address in VLAN 1 for management;

vlan 1
ip address 192.168.1.25 255.255.255.0
no shut
exit

Let’s create VLAN 100 and VLAN 200 on the switch;

vlan 100
name "192-168-100-0/24"
exit
vlan 200
name "192-168-200-0/24"
exit

Let’s add the appropriate edge ports to each VLAN;

interface range gigabitEthernet 1/0/1-12
switchport access vlan 1
exit
interface range gigabitEthernet 1/0/13-24
switchport access vlan 100
exit
interface range gigabitEthernet 1/0/25-36
switchport access vlan 200
exit

Let’s configure ports 45 and 46 as trunk ports and bond them together in channel-group utilizing LACP;

interface gigabitEthernet 1/0/45
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface gigabitEthernet 1/0/46
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

Cisco Catalyst 3750-E – Show Commands

SW-3750-E#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi1/0/45  FA      32768     001e.7e7c.2c00  16s    0x0    0x300A 0x2D    0x3F
Gi1/0/46  FA      32768     001e.7e7c.2c00  27s    0x0    0x300A 0x2E    0x3F

Switch#show mac address-table
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
 All    ffff.ffff.ffff    STATIC      CPU
   1    0004.619e.467e    DYNAMIC     Po1
   1    000c.2964.33f9    DYNAMIC     Po1
   1    000c.29a5.cb54    DYNAMIC     Po1
   1    000f.2095.38d5    DYNAMIC     Po1
   1    0018.01ea.f445    DYNAMIC     Po1
   1    001c.116b.dc6b    DYNAMIC     Po1
   1    001c.116d.1527    DYNAMIC     Po1
   1    001c.116d.15dc    DYNAMIC     Po1
   1    001e.7e7c.2c01    DYNAMIC     Po1
   1    001e.7e7c.2c2d    DYNAMIC     Po1
   1    001e.7e7c.2c2e    DYNAMIC     Po1
   1    001f.d0d0.be2d    DYNAMIC     Po1
   1    0023.ee96.aa21    DYNAMIC     Po1
   1    00a0.f85e.cebd    DYNAMIC     Po1
 100    0024.7f99.84e9    DYNAMIC     Po1
 200    0008.02e4.890a    DYNAMIC     Gi1/0/25
 200    001e.caf3.1db4    DYNAMIC     Po1
Total Mac Addresses for this criterion: 37

You might be asking why didn’t I assign the VLANs to the trunk ports on the Cisco Catalyst 3750-E… well with Cisco switches a trunk port is by default a member of all the VLANs that exist on the switch. So you don’t need to specifically add a VLAN to a trunk port, however, you can override the default behavior by telling the switch to only carry specific VLANs on a specific trunk port – this is called VLAN pruning.

Please feel free to point out any inconsistencies or errors I might have made.

Cheers!

Layer 3 and vPC Recommendations

I was provided a copy of the Nexus 7000 virtual Port-Channel Best Practices & Design Guidelines which clearly indicates on page 25 that routers should not be connected to a vPC link but should instead be connected via a Layer 3 switch port. Here are some bullet points;

• Use separate L3 links to hook up routers to a vPC domain is still standing.
• Don’t use L2 port channel to attach routers to a vPC domain unless you can statically route to HSRP address
• If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2 port-channel for bridged traffic

I was still currious to understand more of the inner-workings.. why didn’t it work or wasn’t it allowed? I only had to flip through the next few slides although I can really say that I completely understand just yet.

1. Packet arrives at R
2. R does lookup in routing table and sees 2 equal paths going north (to 7k1 & 7k2)
3. Assume it chooses 7k1 (ECMP decision)
4. R now has rewrite information to which router it needs to go (router MAC 7k1 or 7k2)
5. L2 lookup happens and outgoing interface is port-channel 1
6. Hashing determines which port-channel member is chosen (say to 7k2)
7. Packet is sent to 7k2
8. 7k2 sees that it needs to send it over the peer-link to 7k1 based on MAC address
9. 7k1 performs lookup and sees that it needs to send to S
10. 7k1 performs check if the frame came over peer link & is going out on a vPC.
11. Frame will only be forwarded if outgoing interface is NOT a vPC or if outgoing vPC doesn’t have active interface on other vPC peer (in our example 7k2)

I’m not embarrassed to say that I followed everything up until step 11. Why exactly is it that frames will only be forwarded if the outgoing interface is NOT a vPC or if the outgoing vPC doesn’t have an active interface on another vPC peer? Isthere anyone that can shed any additional light on this topic?

I’ve never experienced such a restriction in all my years of working with the Avaya (formerly Nortel) Ethernet Routing Switch 8600 and their Split Multilink Trunking (SMLT) technology. I actually have a Cisco 3825 router connected via a SMLT attached Ethernet Routing Switch 5520 (Layer 2) with the Cisco 3825 and the Avaya 8600s all running BGP.

Cheers!

It’s definitely well worth the time to review.

Cheers!

Let me start by saying that we haven’t abandoned Nortel/Avaya as a viable network electronics vendor. When we first started this effort back in January 2009 we initially looked at Brocade, Cisco, HP, Juniper and Nortel. After the initial cut we were left with Brocade, Cisco and Nortel/Avaya. When the final decision was made, back in November 2009, there was still a lot of concern about Avaya’s purchase of Nortel’s Enterprise Division so the decision really came down to Brocade or Cisco. Ultimately the decision was made to go with Cisco. As I commented in the previous post we still have well over 33,000 ports on Nortel/Avaya switches and less than 1,000 ports on Cisco switches. We are still purchasing and installing/deploying Nortel/Avaya switches in our hospitals and business offices .

Nexus 5010 Virtual PortChannel bug?

I’ve been known to tell people that we’re a partner with Nortel/Avaya because of the number of bugs that we identify and help to resolve.  Well if I thought it was going to be any different with Cisco I can throw that idea right out the window as we have identified a bug in the vPC code for the Nexus 5000 switch. We discovered the issue with a pair of Nexus 5010 switches when one of the switches up and died (hardware failure). While trying to troubleshoot the problem we restarted the remaining Nexus 5010 switch only to find that the remaining Nexus 5010 would no longer adopt the Nexus 2148 switches/FEXes after it was rebooted if the peer Nexus 5010 wasn’t online. If we removed the vpc configuration from the portchannel then the FEX would come online. After speaking with Cisco they have created bug CSCth01969 which is supposedly going to be addressed in the next software release.

HP Virtual Connect Flex-10/NIC issues/ESX 4.0 Update 1 Drivers

We also ran into a number of issues with the HP BL490c blades and HP Virtual Connect Flex-10 Interconnect. On a number of blades we had an issue where the ESX installer was unable to locate any network interfaces. The installer would error with something like “The script 32.networking-drivers failed to execute and the installation can not continue.” The solution was to re-apply the NIC firmware after the Virtual Connect profile had been applied to the server. You can find additional detail over at Brian’s blog if interested.

We also ran into another issue when performing our redundancy and high-availability testing. We configured Virtual Connect to use SmartLink, however, when we unplugged the uplinks from the B side interconnect the Virtual Connect Flex-10 interconnect didn’t disable the server side NICs. The solution to this problem was to update the NIC driver that shipped with ESX 4.0 Update 1. You can find the driver here.

Lots of additional stories to come including a comparison of vPC and SMLT.

Cheers!