Field Notice: FN – 72150 – Nexus 9000/3000 Will Fail With SSD Read-Only Filesystem – Power Cycle Required – BIOS/Firmware Upgrade Recommended

The issue was further compounded by some sloppy management, with several switches having unsaved configurations or having crashed and rebooted with unsaved configurations and ultimately inconsistent VPC states. In the short term I ended up deploying the SSD firmware update to all the impacted Cisco Nexus 9000 series switches in my network. I’ll look at performing the recommended software upgrades early next year.

You can setup notifications on the Cisco website to help keep you informed of field notices, software releases and security bulletins.

Anyone else run into this problem?

Cheers!

As many readers will recognize I successfully deployed Avaya now Extreme (formerly Nortel) Ethernet switching solutions at my previous employer for 17 years with great success. The Avaya/Extreme product was extremely cost competitive and provided every feature that we needed to provide a highly reliable network infrastructure to a large healthcare provider. So it shouldn’t surprise anyone that I was more than comfortable looking outside of Cisco’s product offerings.

In the early stages I personally felt that HPE was probably the best positioned to win our business. I had some experience with the HPE/Aruba 3810M and it had worked well in a number of consulting engagements. That was until I received the pricing from Juniper. Juniper literally wiped up the floor and quite literally walked away with our business. We looked at the following products;

• Cisco 3850X
• Cisco 2960X
• Cisco Meraki
• HPE/Aruba 3810M
• Juniper EX4300
• Juniper EX3400

In the end we landed on the Juniper EX4300-48P because it met all of our requirements and enabled us make extremely efficient use of our budget. We’ve deployed about 7 IDFs so for (~ 64 switches all total) connected to a pair of Juniper EX4600s via 10GBaseLR SFPs and we have yet to run into any major problems or issues. We did run into a few problems… but those were quickly fixed with a bag of cotton balls and some rubbing alcohol  – we had to clean the fiber patches.

What did we look at in our selection? Here’s the matrix we ended up creating. If there is some inaccurate information in the table below please post a comment and I’ll be happy to update the data accordingly. I have excluded pricing information, you’ll need to-do your own homework on that front.

Since my current employer believes steadfastly in 100% patching in the closet we usually end up with some very large IDFs and so the ability to have 10 switches in a stack was a large consideration. We also have a number of IDFs with very shallow racks and/or cabinets and that necessitated only looking at solutions that were under 17 inches deep.

That leaves us with a pair of Juniper EX4600s acting as the campus core running OSPF and connecting to the Juniper EX4300-48Ps in the IDFs in a Virtual Chassis configuration. Each IDF is it’s own L2 domain with OSPF routing between the IDF and the campus core. The Juniper EX4600s in turn connect to the Data Center Cisco 6509s. Next year’s project will be to replace the Cisco 6509s that are still in the Data Center.

I hope to put out a sample configuration guide in the coming weeks for the Juniper EX4300-48P.

Cheers!

Upon reviewing the 9.1.7 release notes from Cisco I stumbled over the following entry;

CSCvd78303 – ARP functions fail after 213 days of uptime, drop with error ‘punt-rate-limit-exceeded’

We upgraded a large number of our Cisco ASAs about 213 days ago to 9.1.(7)12 and now it would seem we’ll be upgrading again to 9.1.(7)19 – the actual issue is resolved in 9.1(7)16.

Here’s the text of the bug report;

Symptom:
An ASA, after reaching an uptime of roughly 213 days will fail to process ARP packets leading to a condition where all traffic eventually stops passing through the affected device. Since not all existing ARP entries time out at the same time, not all connections may fail at the same time.
Additional symptoms include:

• ASA does not have ARP entries in its ARP table. show arp is empty
• The output of show asp drop and ASP drop captures indicate a rapidly increasing counter for punt-rate-limit exceeded and the dropped packets are predominantly ARP.

IMAGES WITH FIXES
Images with fixes for this defect will be published as soon as they are available, and posted to the Cisco Software Download center.
Conditions:
This is seen when the ASA’s uptime reaches 213 days.
This problem affects ASA and FTD versions:
ASA version 9.1 releases 9.1(7)8 and higher
ASA version 9.2 releases 9.2(4)15 and higher
ASA version 9.4 releases 9.4(3)5 and higher including 9.4(4)
ASA version 9.5 releases 9.5(3) and higher
ASA version 9.6 releases 9.6(2)1 and higher including 9.6(3)
ASA version 9.7 releases 9.7(1) and higher
FTD version 6.1 releases 6.1.0.1 and higher
FTD version 6.2 releases 6.2.0

Workaround:
Perform a pre-planned reboot of the device before approaching the 213 days (5124 hours) of up time. After the reboot, it will give you another 213 days of up time.
Further Problem Description:
Devices encountering this issue will not receive or respond to ARP packets. This affects not just transient traffic, but also access to the affected device – including Administration access such as SSH, HTTPS and Telnet. Console access is not affected.

If your running a Cisco ASA firewall I would recommend you check to make sure that you won’t be impacted by this bug.

Cheers!

A few years ago I had to work with an ISP that couldn’t understand how my Internet link was being flooded by packets that were being sourced from 127.0.0.1 egressing their network. I had to send them multiple packet traces and tediously explain to them what IP spoofing was and how to block it on ingress into their network backbone.

On Cisco IOS routers we only need to create an extended access-list to identify those packets which don’t have valid source IP addresses. We want to permit any packet which has a valid source IP address from our public IANA IP address block but we want to deny/block any IP packet which has a spoofed source IP address from leaving our network. In the example below I’m using the three TEST networks that are designated by the IETF/IANA for use in documentation. You should substitute the three IP address blocks below with your own public ARIN/APNIC/RIPE IP address assignment.

ip access-list extended bcp38_out
 permit ip 203.0.113.0 0.0.0.255 any 
 permit ip 198.51.100.0 0.0.0.255 any
 permit ip 192.0.2.0 0.0.0.255 any
 deny any any log

Now that we have our access-list we need to apply it to outbound traffic leaving our Internet facing interface.

interface fa0/1
 ...
 ip access-group bcp38_out out
 ...

And that’s it, pretty easy and painless yet too many folks miss this simple step.

Cheers!

We need to generate the following pieces:

1. Generate a private key for this specific use
2. Using the private key generate Certificate Signing Request (CSR)
3. Have the CSR signed by a private or public Certificate Authority which will provide the certificate
4. Upload the private key and signed certificate to your device or system.

Let’s start by creating a directory just for this specific certificate, makes it easier to track all the files we’ll have when we’re complete. In this example I’m going to request a certificate for a Cisco ASA to be used with the Cisco AnyConnect VPN client, vpn.acme.com.

mkdir ~/vpn.acme.com/
cd ~/vpn.acme.com/

Let’s generate a private key, using a key size of 4096 which should future proof us sufficiently.

openssl genrsa -out vpn.acme.com.key 4096

Now let’s generate a SHA 256 certificate request using the private key we generated above.

openssl req -new -sha256 -key vpn.acme.com.key -out vpn.acme.com.csr

We now need to take the certificate request and have that signed by a Certificate Authority. The resulting certificate (filename: vpn.acme.com.crt) will need to be installed along with the private key onto the appliance or device that we’re generating the certificate for.

Since we’re working with a Cisco ASA we need to combine the private key, certificate and any intermediate certificate authorities into a single PKCS12 file so we can upload that file into our Cisco ASA. Again we’ll use OpenSSL for this task and it’s pretty easy. (You’ll be prompted to set a password on the file, make sure you don’t forget it because you’ll need it to upload the file into the Cisco ASA).

openssl pkcs12 -export -in vpn.acme.com.crt -inkey vpn.acme.com.key \
        -certfile public-intermediate-ca.crt -out vpn.acme.com_bundle.p12

Now we can upload the bundle file (vpn.acme.com_bundle.p12) to the Cisco ASA.

Cheers!

The laptop would often connect to the SSID but the laptop would be unable to get a webpage to render with all IP traffic essentially stalling. ICMP ping times would jump from 1 ms to 3,900 ms with multiple dropped packets scattered all about the constant ping. Without any load you could occasionally get 1 ms response times for a couple of minutes at a time but the instant you opened a web page the traffic would stall and the ICMP pings would start timing out.

The Intel engineer that was assisting me provided the hint, letting me know that Cisco IT had actually stumbled across this very same issue the week earlier internally with their own employees. Cisco had intentionally disabled A-MPDU on their WLCs, the workaround was to enable A-MPDU for 802.11n on their WLCs. I went ahead and checked our WLCs and sure enough we also had A-MPDU disabled – not exactly sure who or why it was disabled.

802.11n Status: 
    A-MPDU Tx: 
        Priority 0............................... Disabled 
        Priority 1............................... Disabled 
        Priority 2............................... Disabled 
        Priority 3............................... Disabled 
        Priority 4............................... Disabled 
        Priority 5............................... Disabled 
        Priority 6............................... Disabled 
        Priority 7............................... Disabled 
        Aggregation scheduler.................... Enabled 
        Frame Burst.............................. Automatic 
            Realtime Timeout..................... 10 
    A-MSDU Tx: 
        Priority 0............................... Enabled 
        Priority 1............................... Enabled 
        Priority 2............................... Enabled 
        Priority 3............................... Enabled 
        Priority 4............................... Enabled 
        Priority 5............................... Enabled 
        Priority 6............................... Disabled 
        Priority 7............................... Disabled 
    Rifs Rx ..................................... Enabled 
    Guard Interval .............................. Any 

I used the following CLI commands to enable A-MPDU; (note that I had to temporarily disable the 802.11a network to make the change – you’ll want to schedule this off-hours)

config 802.11a disable 
y 
config 802.11a 11nsupport a-mpdu tx priority 0 enable 
config 802.11a 11nsupport a-mpdu tx priority 1 enable 
config 802.11a 11nsupport a-mpdu tx priority 2 enable 
config 802.11a 11nsupport a-mpdu tx priority 3 enable 
config 802.11a 11nsupport a-mpdu tx priority 4 enable 
config 802.11a 11nsupport a-mpdu tx priority 5 enable 
config 802.11a enable 

Why doesn’t the Intel AC 8260 wireless adapter negotiate using A-MSDU?

I hope to be able to bring you that answer from either Cisco or Intel.

I hope you enjoyed the article Tim.

Cheers!

Update: December 7, 2016

Intel has released a new driver for the AC 8260 that is designed to address the issue.
https://downloadcenter.intel.com/download/26465/Intel-PROSet-Wireless-Software-and-Drivers-for-Windows-10
https://downloadcenter.intel.com/download/26469/Intel-PROSet-Wireless-Software-and-Drivers-for-IT-Admins

I’m currently testing the driver but haven’t had enough time to comment yet.

On Wednesday morning I had two Cisco 5508 Wireless LAN Controllers both crash with a “Bonjour_Process_Task” taking too much cpu: 100% error message. It turns out that this is a known issue (CSCux78464 WLC crashes in Process Bonjour_Process_Task) that is resolved in 8.0.135.1, an engineering release which you need to contact Cisco TAC to obtain. If that wasn’t enough excitement for the morning I quickly noticed that of 120 APs that we usually have connected to the WLC we only had about 70 APs connected.  A quick examination of the debug logs (debug capwap errors enable) showed that multiple APs were failed to join the controller with messages like “Discarding non-ClientHello Handshake or DTLS encrypted packet” and “DTLS session is not established”. A quick call to Cisco TAC revealed that there are built-in certificates into the APs that can expire over time and that’s what had essentially happened. The certificates had expired since the APs had last joined the WLC and now that the certificates were expired they were not able to join the controller. Thankfully there’s a command in the CLI to ignore the certificate expiration;

config ap cert-expiry-ignore mic enable

With that command configured on the WLC the APs starting joining the controller and all was well again.

The field notice from Cisco providing all the details can be found here.

Cheers!

I noticed while working on setting up this lab that the MST digest between the Cisco and HP switches didn’t match. After some quick research it appears that the Cisco 2950s I have in the lab operate with a pre standard MST operation. Other Cisco switches identify them as such and are interoperable but you may have issues with third-party devices that are expecting the 802.1s standard. You can see both digests from the Cisco 3750 below and they match both the Cisco 2950 and the HP 2810 switches.

C3750-SW1#show spanning-tree mst configuration digest
Name      [AcmeNetworks]
Revision  1     Instances configured 3
Digest          0x6DA4B50C4FD587757EEF0356753605E1
Pre-std Digest  0x421D7D23BF9562A0C35E46CA1BE8A75C

Example Topology

You’ll notice the HP switches at the bottom of the diagram. It was pretty straight forward but here’s what I needed to do.

Cisco Catalyst 2950 Switch 1 & 2

First we needed to configure the ports on the Cisco 2950s that would be connected to the HP switches. I used Port Channel 3 for this and enabled LACP;

interface fas0/15
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active

interface fas0/16
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active

HP 2810 Switch 1 & 2

Now we need to configure the HP switches, VLANs, IP addressing, ports, trunking, MST, etc;

vlan 100
name "192-168-100-0/24"
vlan 200
name "192-168-200-0/24"

vlan 100
ip address 192.168.100.70 255.255.255.0
exit

spanning-tree
spanning-tree config-name "AcmeNetworks"
spanning-tree config-revision 1
spanning-tree instance 1 vlan 100
spanning-tree instance 2 vlan 200

trunk 1,13 trk1 lacp
trunk 23,24 trk2 lacp

vlan 100 tagged trk1
vlan 200 tagged trk1

vlan 100 tagged trk2
vlan 200 tagged trk2

That’s all well and good but I’m sure you want to see the output… is it working as expected? Well let’s check it out.

Cisco Catalyst 2950 Switch 1

We can see from the data below that LACP has established to the HP switch and Spanning Tree is working as expected;

C2950-SW1#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0064.40cf.4d80  24s    0x3     0x102    0x3D
Fa0/2     SA      32768     0064.40cf.4d80  17s    0x3     0x103    0x3D

Channel group 2 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/31    SA      32768     0018.ba8e.4a40  22s    0x2     0x1F     0x3D
Fa0/33    SA      32768     0018.ba8e.4a40   3s    0x2     0x21     0x3D

Channel group 3 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/15    SA      0         0026.f1df.f400  21s    0x32    0x18     0x3D
Fa0/16    SA      0         0026.f1df.f400  21s    0x32    0x17     0x3D

C2950-SW1#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    16384
             Address     3475.c732.a400
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0019.2faa.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p
Po3              Desg FWD 100000    128.67   P2p Bound(RSTP)


MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    16385
             Address     54e0.322a.d441
             Cost        120000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0019.2faa.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p
Po3              Boun FWD 100000    128.67   P2p Bound(RSTP)


MST02
  Spanning tree enabled protocol mstp
  Root ID    Priority    16386
             Address     0064.40cf.4d80
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0019.2faa.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p
Po3              Boun FWD 100000    128.67   P2p Bound(RSTP)

C2950-SW1#show spanning-tree mst configuration digest
Name      [AcmeNetworks]
Revision  1
Instance  Vlans mapped
--------  ---------------------------------------------------------------------
0         1-99,101-199,201-4094
1         100
2         200
-------------------------------------------------------------------------------
Digest    421D7D23BF9562A0C35E46CA1BE8A75C

Cisco Catalyst 2950 Switch 2

We can see from the data below that LACP has established to the HP switch and Spanning Tree is working as expected;

C2950-SW2#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
Channel group 1 neighbors

Partner's information:

LACP port Oper Port Port
Port Flags Priority Dev ID Age Key Number State
Fa0/47 SA 32768 0064.40cf.4d80 2s 0x4 0x130 0x3D
Fa0/48 SA 32768 0064.40cf.4d80 25s 0x4 0x131 0x3D

Channel group 2 neighbors

Partner's information:

LACP port Oper Port Port
Port Flags Priority Dev ID Age Key Number State
Fa0/31 SA 32768 0019.2faa.49c0 27s 0x2 0x1F 0x3D
Fa0/33 SA 32768 0019.2faa.49c0 19s 0x2 0x21 0x3D

Channel group 3 neighbors

Partner's information:

LACP port Oper Port Port
Port Flags Priority Dev ID Age Key Number State
Fa0/15 SA 0 0026.f1e1.41a0 29s 0x32 0x17 0x3D
Fa0/16 SA 0 0026.f1e1.41a0 0s 0x32 0x18 0x3D

C2950-SW2#show spanning-tree

MST00
Spanning tree enabled protocol mstp
Root ID Priority 16384
Address 3475.c732.a400
Cost 0
Port 65 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address 0018.ba8e.4a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100000 128.65 P2p
Po2 Desg FWD 100000 128.66 P2p
Po3 Desg FWD 100000 128.67 P2p Bound(RSTP)

MST01
Spanning tree enabled protocol mstp
Root ID Priority 16385
Address 54e0.322a.d441
Cost 120000
Port 65 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 0018.ba8e.4a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100000 128.65 P2p
Po2 Desg FWD 100000 128.66 P2p
Po3 Boun FWD 100000 128.67 P2p Bound(RSTP)

MST02
Spanning tree enabled protocol mstp
Root ID Priority 16386
Address 0064.40cf.4d80
Cost 100000
Port 65 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32770 (priority 32768 sys-id-ext 2)
Address 0018.ba8e.4a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100000 128.65 P2p
Po2 Desg FWD 100000 128.66 P2p
Po3 Boun FWD 100000 128.67 P2p Bound(RSTP)

C2950-SW2# show spanning-tree mst configuration digest
Name [AcmeNetworks]
Revision 1
Instance Vlans mapped
-------- ---------------------------------------------------------------------
0 1-99,101-199,201-4094
1 100
2 200
-------------------------------------------------------------------------------
Digest 421D7D23BF9562A0C35E46CA1BE8A75C

HP 2810 Switch 1

HP-SW1# show lacp

                           LACP

   PORT   LACP      TRUNK     PORT      LACP      LACP
   NUMB   ENABLED   GROUP     STATUS    PARTNER   STATUS
   ----   -------   -------   -------   -------   -------
   1      Active    Trk1      Up        Yes       Success
   13     Active    Trk1      Up        Yes       Success
   23     Active    Trk2      Up        Yes       Success
   24     Active    Trk2      Up        Yes       Success


HP-SW1# show cdp neighbors

 CDP neighbors information

  Port Device ID                     | Platform                     Capability
  ---- ----------------------------- + ---------------------------- -----------
  1    00 26 f1 e1 41 a0             | ProCurve J9021A Switch 28... S
  13   00 26 f1 e1 41 a0             | ProCurve J9021A Switch 28... S
  23   C2950-SW1                     | Cisco Internetwork Operat... S
  24   C2950-SW1                     | Cisco Internetwork Operat... S

HP-SW1# show spanning-tree

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes
  Force Version : MSTP-operation
  IST Mapped VLANs : 1

  Switch MAC Address : 0026f1-dff400
  Switch Priority    : 32768
  Max Age  : 20
  Max Hops : 20
  Forward Delay : 15

  Topology Change Count  : 332
  Time Since Last Change : 53 mins

  CST Root MAC Address : 3475c7-32a400
  CST Root Priority    : 16384
  CST Root Path Cost   : 200000
  CST Root Port        : Trk2

  IST Regional Root MAC Address : 0026f1-dff400
  IST Regional Root Priority    : 32768
  IST Regional Root Path Cost   : 0
  IST Remaining Hops            : 20

  Root Guard Ports :
  TCN Guard Ports  :
  Protected Ports :
  Filtered Ports :

                  |           Prio             | Designated    Hello
  Port  Type      | Cost      rity  State      | Bridge        Time  PtP Edge
  ----- --------- + --------- ----- ---------- + ------------- ----- --- ----
  2     100/1000T | Auto      128   Disabled   |
  3     100/1000T | Auto      128   Disabled   |
  4     100/1000T | Auto      128   Disabled   |
  5     100/1000T | Auto      128   Disabled   |
  6     100/1000T | Auto      128   Disabled   |
  7     100/1000T | Auto      128   Disabled   |
  8     100/1000T | Auto      128   Disabled   |
  9     100/1000T | Auto      128   Disabled   |
  10    100/1000T | Auto      128   Disabled   |
  11    100/1000T | Auto      128   Disabled   |
  12    100/1000T | Auto      128   Disabled   |
  14    100/1000T | Auto      128   Disabled   |
  15    100/1000T | Auto      128   Disabled   |
  16    100/1000T | Auto      128   Disabled   |
  17    100/1000T | Auto      128   Disabled   |
  18    100/1000T | Auto      128   Disabled   |
  19    100/1000T | Auto      128   Disabled   |
  20    100/1000T | Auto      128   Disabled   |
  21    100/1000T | Auto      128   Disabled   |
  22    100/1000T | Auto      128   Disabled   |
  Trk1            | 20000     64    Forwarding | 0026f1-dff400 2     Yes No
  Trk2            | 200000    64    Forwarding | 00192f-aa49c0 2     Yes No

HP-SW1# show spanning-tree instance 1

 MST Instance Information

  Instance ID : 1
  Mapped VLANs : 100

  Switch Priority         : 32768

  Topology Change Count   : 39
  Time Since Last Change  : 53 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 0
  Regional Root Port        : This switch is root
  Remaining Hops            : 20
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Designated Forwarding 0026f1-dff400
  Trk2            200000    128      Master     Forwarding 0026f1-dff400

HP-SW1# show spanning-tree instance 2

 MST Instance Information

  Instance ID : 2
  Mapped VLANs : 200

  Switch Priority         : 32768

  Topology Change Count   : 38
  Time Since Last Change  : 53 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 0
  Regional Root Port        : This switch is root
  Remaining Hops            : 20
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Designated Forwarding 0026f1-dff400
  Trk2            200000    128      Master     Forwarding 0026f1-dff400

HP-SW1# show spanning-tree mst-config

 MST Configuration Identifier Information

  MST Configuration Name : AcmeNetworks
  MST Configuration Revision : 1
  MST Configuration Digest : 0x6DA4B50C4FD587757EEF0356753605E1

  IST Mapped VLANs : 1

  Instance ID Mapped VLANs
  ----------- ---------------------------------------------------------
  1           100
  2           200

HP 2810 Switch 2

HP-SW2# show lacp

                           LACP

   PORT   LACP      TRUNK     PORT      LACP      LACP
   NUMB   ENABLED   GROUP     STATUS    PARTNER   STATUS
   ----   -------   -------   -------   -------   -------
   1      Active    Trk1      Up        Yes       Success
   13     Active    Trk1      Up        Yes       Success
   23     Active    Trk2      Up        Yes       Success
   24     Active    Trk2      Up        Yes       Success


HP-SW2# show cdp neighbors

 CDP neighbors information

  Port Device ID                     | Platform                     Capability
  ---- ----------------------------- + ---------------------------- -----------
  1    00 26 f1 df f4 00             | ProCurve J9021A Switch 28... S
  13   00 26 f1 df f4 00             | ProCurve J9021A Switch 28... S
  23   C2950-SW2                     | Cisco Internetwork Operat... S
  24   C2950-SW2                     | Cisco Internetwork Operat... S

HP-SW2# show spanning-tree

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes
  Force Version : MSTP-operation
  IST Mapped VLANs : 1

  Switch MAC Address : 0026f1-e141a0
  Switch Priority    : 32768
  Max Age  : 20
  Max Hops : 20
  Forward Delay : 15

  Topology Change Count  : 65
  Time Since Last Change : 66 mins

  CST Root MAC Address : 3475c7-32a400
  CST Root Priority    : 16384
  CST Root Path Cost   : 200000
  CST Root Port        : Trk1

  IST Regional Root MAC Address : 0026f1-dff400
  IST Regional Root Priority    : 32768
  IST Regional Root Path Cost   : 20000
  IST Remaining Hops            : 19

  Root Guard Ports :
  TCN Guard Ports  :
  Protected Ports :
  Filtered Ports :

                  |           Prio             | Designated    Hello
  Port  Type      | Cost      rity  State      | Bridge        Time  PtP Edge
  ----- --------- + --------- ----- ---------- + ------------- ----- --- ----
  2     100/1000T | Auto      128   Disabled   |
  3     100/1000T | Auto      128   Disabled   |
  4     100/1000T | Auto      128   Disabled   |
  5     100/1000T | Auto      128   Disabled   |
  6     100/1000T | Auto      128   Disabled   |
  7     100/1000T | Auto      128   Disabled   |
  8     100/1000T | Auto      128   Disabled   |
  9     100/1000T | Auto      128   Disabled   |
  10    100/1000T | Auto      128   Disabled   |
  11    100/1000T | Auto      128   Disabled   |
  12    100/1000T | Auto      128   Disabled   |
  14    100/1000T | Auto      128   Disabled   |
  15    100/1000T | Auto      128   Disabled   |
  16    100/1000T | Auto      128   Disabled   |
  17    100/1000T | Auto      128   Disabled   |
  18    100/1000T | Auto      128   Disabled   |
  19    100/1000T | Auto      128   Disabled   |
  20    100/1000T | Auto      128   Disabled   |
  21    100/1000T | Auto      128   Disabled   |
  22    100/1000T | Auto      128   Disabled   |
  Trk1            | 20000     64    Forwarding | 0026f1-dff400 2     Yes No
  Trk2            | 200000    64    Blocking   | 0018ba-8e4a40 2     Yes No

HP-SW2# show spanning-tree instance 1

 MST Instance Information

  Instance ID : 1
  Mapped VLANs : 100

  Switch Priority         : 32768

  Topology Change Count   : 11
  Time Since Last Change  : 66 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 20000
  Regional Root Port        : Trk1
  Remaining Hops            : 19
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Root       Forwarding 0026f1-dff400
  Trk2            200000    128      Alternate  Blocking   0026f1-e141a0

HP-SW2# show spanning-tree instance 2

 MST Instance Information

  Instance ID : 2
  Mapped VLANs : 200

  Switch Priority         : 32768

  Topology Change Count   : 10
  Time Since Last Change  : 66 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 20000
  Regional Root Port        : Trk1
  Remaining Hops            : 19
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Root       Forwarding 0026f1-dff400
  Trk2            200000    128      Alternate  Blocking   0026f1-e141a0

HP-SW2#  show spanning-tree mst-config

 MST Configuration Identifier Information

  MST Configuration Name : AcmeNetworks
  MST Configuration Revision : 1
  MST Configuration Digest : 0x6DA4B50C4FD587757EEF0356753605E1

  IST Mapped VLANs : 1

  Instance ID Mapped VLANs
  ----------- ---------------------------------------------------------
  1           100
  2           200

Cheers!

We had presentations from Skyport Systems, NetScout (formerly Fluke), Big Switch, Silver Peak, Dell, Cisco and Citrix. The folks over at Gestalt IT brought together a who’s who list of technology and networking professionals, and I was honored to be included among them. The discussions covered a wide breadth of topics and technology. Over the next few weeks I hope to find the time to post some of my thoughts and perspective on each of the presentations.

It was great to meet Brandon, Dominik, Ethan, Greg, Jason, John, Jon, Jordan, Matt, Phil, Terry, Stephen and Tom. If you missed any of the presentations you can see them again by visiting the Tech Field Day website and clicking on the specific presentations made at Networking Field Day 11.

I had hoped to be writing this from the plane but I’m currently writing this from the Biltmore Hotel and Suites since my return flight was canceled. I’ll hopefully be flying out tomorrow via Chicago and then on to Philadelphia. I’m hearing that the snow is really falling and blowing in the Philadelphia suburbs thanks to Winter Storm Jonas. The forecast is for 18″-24″ of snow for the Philadelphia area. I’m not sure I would describe it as a snowmageddon, but there’s definitely going to be a lot of snow to deal with whenever I return to Philadelphia. I can only imagine what the economy parking lot will look like at the Philadelphia airport.

Stay safe everyone!

Cheers!

Within the past six months my employer opened a new 1 million square foot distribution facility here in southeastern Pennsylvania. It’s been an exciting challenge for me personally. One of the bigger challenges has been getting the wireless network tuned in since the building opened essentially empty and now almost 4 months later is filled to the gills with product as we prepare for the upcoming holiday season. It’s about 45′ to the ceiling framing joists so I had our integrator mount the APs (Cisco AP 2702e) to 6′ of electrical conduit hanging from the ceiling. This way the APs are a little closer to the intended coverage space yet they are high enough to avoid being hit by the many fork lifts and other vehicles that traverse the distribution center.

Weekly I review the power settings and RF coverage as the building literally fills up with product…which attenuates the signal from the wireless APs little by little. The majority of my WiFi experience is in hospitals, very old buildings and corporate office space where the RF signal only covers about 40-50′ at max and sometimes much smaller (hospitals). The biggest surprise in this facility is the propagation of the RF signal, there’s literally too much RF signal. I’ve had to turn off quite a few AP radios especially in the 2.4Ghz (802.11b/g) band to keep the channel interference to a minimum.

While RF coverage issues are no surprise to anyone working in the industry I have had an interesting time tracking down an idle timeout issue between the Motorola handheld scanners and the IBM AS400 host which runs our warehouse inventory system. The default idle user timeout on the Cisco WLC 5508 is 5 minutes, so I had to changed that parameter to 43200 seconds (12 hours). I also changed the session timeout to 86400 seconds (24 hours). Our employee shifts run 10 hours so these settings looked like they would work – and they do work, except they didn’t work for everyone. In my initial testing I found that I could login from one side of the distribution center, walk to the other side of the distribution center (6 minute walk – it’s that big) and I could pickup my session where I left off without issue even with the handheld going to sleep as I physically moved through the distribution center. I thought the problem was solved and moved onto other issues only to eventually hear that some users were still reporting that they were being “kicked out” – but not everyone.

It was a very specific set of users… those working the receiving dock and the wholesale racking. What was the common denominator? These users would occasionally put down their Motorola handhelds for upwards of 10-15 minutes as they unloaded a truck or prepared to put away a pallet. When they tried to use the handheld it would prompt that their AS400 session had ended and they would need to log in again.

I ran a few tests from the office space in the distribution center and quickly determined that if I was idle longer than 12 minutes I would generally get disconnected from my AS400 session. Was this issue coming from the wireless network or the AS400 host itself? I setup a quick and easy control test. I enabled telnet on one of my Linux jump boxes and established a connection from my test MC9190. After logging into my Linux jumpbox I sat the device on my desk for 25 minutes with no input from me. The Motorola MC9190 went to sleep in about 30 seconds, 25 minutes later I picked up the device, waited for it to power up and connect to the network and found my telnet session still working without issue so this test essentially eliminated the wireless network.

It turns out that the AS400 uses a TCP keepalive to determine if a user is still connected to the system. I’m guessing that since the handheld is sleeping it’s not responding to the keepalive packets so the host system is eventually closing the connection to the handheld. I have some inquires into our AS400 team and IBM to validate my theory.

Cheers!

Update: Sunday October 25, 2015

We were able to update the session keep-alive parameter on the IBM AS400 host. This value defaults to 10 minutes. So every 10 minutes the host will check if the session is still valid by sending a keep-alive packet. If the IBM AS400 host doesn’t get a response to the check it will logoff the session. We increased this value to 20 minutes and that’s helped greatly.

The WS5100 would intermittently come under extreme load for 5-30 minutes, so much load that ultimately the entire wireless network would collapse as the Access Ports started experiencing watchdog resets and would just continually reboot themselves. This problem would come and go throughout the day or night, we could go 12 hours without an issue and then go the next 12 hours with issues every 30 minutes. The problem was affecting both the primary and secondary WS5100 so I eliminated the hardware almost out of the gate. I have first hand experience running v3.3.5.0-002R software on a large number of WS5100s and have never had an issue with that software release so I really didn’t suspect the software. This wireless solution had been in place for more than 18 months without any major issues or problems. The local engineers reported that there had been no changes, no new devices. So what was causing this problem? I immediately suspected an external catalyst but how would I find it?

As with most highly technical problems it wasn’t until I could get my hands on some packet traces and I had time to dissect those packet traces that I could start to fully understand and comprehend what was actually going on.

Topology

A pair of Motorola WS5100 Wireless LAN Switches with 30 AP300 running software release v3.3.5.0-002R in a cluster configuration with one running as primary and the other running as secondary. The network was comprised of a single Cisco Catalyst 4500 with around ten individual Cisco Catalyst 2960S switches at the edge each trunked to the core in a simple hub and spoke design. The entire network was one single flat VLAN. The WS5100s were attached to the Cisco Catalyst 4500 via a single 1Gbs interface, one arm router style. The peak number of wireless devices was around 200, the total number of MAC addresses on the network was around 525 (this includes the wireless devices).

Symptoms

The initial problem report centered around poor wireless performance and sure enough I quickly found 30-40% packet loss while just trying to ping the WS5100. When I finally got logged into the WS5100 I could see that the CPU was running at 100%. The SYSLOG data showed me that the APs were rebooting because of watchdog timeouts. PTRG was showing me that here was a huge traffic surge being received from the WS5100. I quickly realized that the traffic spikes in the graph correspond to events that users were experiencing problems.

Packet Traces

I directed the team to setup a SPAN port to capture the traffic that was flowing between the WS5100 and the Cisco Catalyst 4500 switch. This would provide me a better idea of what was actually on the wire and might provide a clue as to what was transpiring. The team setup Wireshark to continually capture to disk using a 100MB file size and allowing the file to wrap 10 times for a total of 1GB of captured data. The next time the problem occurred I was alerted within 15 minutes by the help desk and users but I found that we missed the start of the event. There was so much traffic Wireshark only had the past 3 minutes available on disk so we had to increase the filesize to 300MB and the number of wrap files to 25 giving us a total capacity of 7.5GB. That configuration would eventually allow me to capture the initial events along with the time needed to get to the laptop and copy the data before it was overwritten. While I waited for the problem to occur I took to setting up SWATCH to alert myself and the team when the problem started so we could quickly gather all the data points during the start of the event.

Using the data from the packet traces we were able to identify and locate two HP desktops that were apparently intermittently flooding the network with ICMPv6 Multicast Listener Reports.

We removed those HP desktops from the network and everything has been stable since.

Analysis

Here’s the current working theory which I believe is fairly accurate. The HP desktops were intermittently flooding the network with ICMPv6 Multicast Listener Reports. Those packets were reaching the WS5100 and because the network at this location is a single flat VLAN the WS5100 needs to bridge those packets over to the wireless network. It does this by encapsulating them in MiNT in a fashion very similar to CAPWAP  or LWAPP. The issue here is the number of packets and the number of access points or access ports. In this case we had 30 APs connected to the WS5100 so let’s do some rough math;

41,000 ICMPv6 Multicast packets * 2 HP desktops = 82,000 packets * 30 APs = 2,460,000 packets

This explains the huge amount of traffic the WS5100 is transmitting. For every ICMPv6 Multicast packet (or broadcast packet for that matter) received by the WS5100, it needs to encapsulate and send a copy of that packet to each and every AP. If there are 30 APs then the WS5100 needs to copy each and every packet 30 times. Now multiply that by the number of ICMPv6 packets that were being received by the WS5100 and you have a recipe for disaster.

A quick search of Google will reveal a number of well documented issues with Intel NICs.

The HP desktops turned out to be HP ProDesk 600 G1s running Windows 7 SP1 with Intel I217-LM NICs driver v12.10.30.5890 with sleep and WoL enabled.

Summary

There were a few lessons learned here;

1. The days of the single flat network are gone. It’s very important to follow best practice when designing and deploying both wired and wireless infrastructures. In this case if the wireless infrastructure had dedicated VLANs both for the wireless client traffic and for the AP traffic this problem would have never impacted the WS5100. It may have impacted the Cisco Catalyst 4500 somewhat but it wouldn’t have caused the complete collapse of the wireless infrastructure. Unfortunately in this case everything was on VLAN 1, wired clients, APs, wireless clients, servers, IP phone systems, routers, everything.
2. The filtering of IPv6 along with Multicast and broadcast traffic from the wireless infrastructure is especially important. I posted back in September 2013 how to filter IPv6, multicast and broadcast packets from a Motorola RFS7000, the same applies to the WS5100. Unless you are leveraging IPv6  in your infrastructure, or have some special multicast applications you should definitely look into filtering this traffic from your wireless network.
3. Validate those desktop and laptop images, especially the NIC drivers and WNIC drivers. In the early days of 802.1x I can remember documenting a long list of driver versions and Microsoft hotfixes required for Microsoft Windows XP (pre SP2) in order to get 802.1x authentication (Zero Wireless Configuration) to work properly.

Conclusion

Wireshark saved this network engineer’s holiday – Thanks!

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 27 of 30. All the posts can be viewed from the 30in30 tag.

In this specific case all the routing was being performed by a number of high-end Cisco ASA firewalls which didn’t have PIM routing configured or enabled so I took the easy approach of just disabling IGMP snooping across the Cisco Catalyst 6504 and 4948 switches and the problem was solved. The cleaner solution would have been to setup an Mutlicast Router (mrouter) on the VLAN to properly handle all the IGMP requests and reports.

As pointed out by a colleague you can use a great little Python script written by RedHat for testing Multicast on your Linux servers.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 3 of 30. All the posts can be viewed from the 30in30 tag.

Reference;
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/68131-cat-multicast-prob.html
http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/route_multicast.html

When a web browser opens a connection to a web server it will generally send multiple requests across a single TCP connection similar to the figure below. Occasionally some browsers will even utilize HTTP pipelining if both the server and browser support that feature, sending multiple requests without waiting for the corresponding TCP acknowledgement.

The majority of load balancers, including the Cisco ACE 4710 and the A10 AX ADC/Thunder, will look at the first request in the TCP connection and apply the load-balancing metric and forward the traffic to a specific real server in the VIP. In order to speed the processing of future requests the load balancer will forward all traffic in that connection to the same real server in the VIP. This generally isn’t a problem if there’s only a single user associated with a TCP connection.

Akamai will attempt to optimize the number of TCP connections from their edge servers to your origin web servers by sending multiple requests from different users all over the same TCP connection. In the example below there are requests from three different users but it’s been my experience that you could see requests for dozens or even hundreds of users across the same TCP connection.

And here lies the problem, the load balancer will only evaluate the first request in the TCP connection, all subsequent requests will be sent to the same real server leaving some servers over utilized and others under utilized.

Thankfully there are configuration options in the majority of load balancers to work around this problem and instruct the load balancer to evaluate all requests in the TCP connection independently.

A10 AX ADC/Thunder

strict-transaction-switch

Cisco ACE 4710

parameter-map type http HTTP_PARAMETER_MAP
  persistence-rebalance strict

With the configuration change made now every request in the TCP connection is evaluated and load-balanced independently resulting in a more even distribution across the real servers in the farm.

In this scenario I’m using HTTP cookies to provide session persistence and ‘stickiness’ for the user sessions. If your application is stateless then you don’t really need to worry that a user lands on the same real server for each and every request.

Cheers!

Image Credit: topfer

References;
LogNormal – http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/

Cheers!

Image Credit: Jaylopez

I decided to write about this topic since I recently encountered operational “difficulties” with both my prior employer and my current employer that involved the same near identical mistake of a network engineer accidentally overwriting the list of allowed VLANs. In the most recent case it was a simple oversight on the engineer’s part and the problem was quickly corrected. On the prior case the engineer had issued the command “no switchport trunk allowed vlan x” which seems to have given NXOS a bit of a fit. The ports that were in that vPC needed to be shutdown and then enabled to clear what appeared to be a software bug. While the running-config indicated that the VLANs were being trunked on the ports, the MAC/FDB table had no entries of those VLANs on the affected ports.

I strongly recommend that folks prune VLANs that aren’t being used from their trunks, however, you need to be very careful with how you add and/or remove VLANs from the list once the trunk is up and running.

In the past I’ve seen folks accidentally overwrite the VLAN allowed add list by using the “switchport trunk allowed vlan” command. Look at this sample configuration;

interface port-channel2
  description VPC_CISCO_NEXUS_5010
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 150-154
  switchport trunk allowed vlan add 155
  mtu 9216
  vpc 2

Now let’s say we wanted to add VLAN 156 and we using the following command, taking the allowed VLAN list and adding VLAN 156;

switchport trunk allowed vlan 152-154,156

The problem with this is that we just missed the fact that VLAN 155 was also on that trunk and we just removed it from the trunk with the that previous command.

The morale of the story – be careful when you add/remove VLANs from trunk ports and make sure you use the switchport trunk allow vlan add|remove command.

Cheers!

Image Credit: Roger Kirby

Articles

It’s been mostly dead all day… by Amy Engineer – Amy recalls for us her trials and tribulations as she battled with a demonic Cisco 7945 IP phone. I’ve had many similar adventures but with Avaya IP phones, so I feel like I can really relate to Amy’s experiences. I can’t begin to count the number of times that Philippe JOUNIN’s Tftpd32 has saved my bacon.

Minimal Kickstart File for Red Hat Enterprise Linux 6, CentOS 6, Oracle Linux 6 Virtual Machines by Bob Plankers – These days I wouldn’t personally be caught dead not using kickstart files but if you haven’t taken the time yet to build one yourself or still don’t know how they work, Bob provides a great tutorial with all the explanation and detail you’ll need to be successful.

8 Things I Have Learned From 20 Years of Data Networking by Greg Ferro – Greg provides some wonderful insights, beyond the technical realm, to help everyone who’s involved with Information Technology. I’m personally working on step 8, Work on Personal Productivity. I’m a very productive individual and I’m very organized but I often feel like I spend too much time bringing the two together so I’m working on my personal work flows trying to optimize how I work to save time and energy.

Installing the Junos EZ Library – Easy SDN Part 1 and Getting Busy With the Junos EZ Library by John Herbert – If you are at all interested in how to interface with Junos utilizing the Junos EZ library written by Jeremy Schulman then you need to check out these post by John. He quite literally takes you through the entire process and shows you how to get it working. I just recently started working with a Juniper EX-2200C so I might have to try it out myself.

Software Releases

• Avaya Virtual Services Platform 9000 v3.3.5.0 – Release Notes
• Avaya Ethernet Routing Switch 8800/8600 v7.2.10.1 – Release Notes
• Avaya Ethernet Routing Switch 5000 series v6.3.2 – Release Notes
• Avaya Ethernet Routing Switch 4000 series v5.6.4 – Release Notes
• Avaya Ethernet Routing Switch 4000 series v5.7 – Release Notes
• Avaya SIP Software Release 4.4 for 11xx/12xx IP Deskphones – Release Notes – Product Correction Notice

The SNMP problems I wrote about in the post entitled, Avaya Ethernet Routing Switch 4800 – Part 2, have been resolved in software release 5.6.4 for the Avaya Ethernet Routing Switch 4000 series. The problem was identified in the release notes as follows, “Snmpgetnext for if Index, ifInOctets and ifType returned incorrect ifInOctets data (WI01118979)”

Let me just take a second to point out two new features in software release 5.7;

EDM inactivity time out
A session becomes inactive if there is no interaction with the EDM interface for more than 15 minutes. After the session becomes inactive, you must login again with your user name and password. Using the ACLI command edm inactivity-timeout, you can configure the time period for which an EDM session remains active.

Thank god they’ve finally made this timer configurable!!!

FastEthernet replaced with Ethernet
The keyword FastEthernet is replaced with Ethernet in all the ACLI commands. For compliance, the old commands containing FastEthernet keyword are hidden, and you can configure using the keyword

I can only tell you the looks I’d get from junior engineers trying to explain to them the differences between Cisco’s IOS, NXOS and then Avaya’s ACLI. They would tell me, “but it’s not a Fast Ethernet interface”, and I would tell them, “your right but you still need to use FastEthernet in your CLI input.”

Cheers!

Before the BGP Soft Reset Enhancement feature, a soft reset for inbound routing table updates was performed by entering the neighbor soft-reconfiguration router configuration command. This command was used to configure the local BGP router to store all received (inbound) routing policy updates. However, this method uses too much memory because inbound updates are not modified and is not recommended.

I’m guessing this new feature had a significant impact for anyone taking a full Internet BGP table?

Cheers!

Image Credit: Earth 3D by Jan

Switch interfaces exchange PAgP packets only with partner interfaces configured in the auto or desirable modes. Switch interfaces exchange LACP packets only with partner interfaces configured in the active or passive modes. Interfaces configured in the on mode do not exchange PAgP or LACP packets.
…
Both the auto and desirable PAgP modes allow interfaces to negotiate with partner interfaces to determine if they can form an EtherChannel based on criteria such as interface speed and, for Layer 2 EtherChannels, trunking state and VLAN numbers.
…
Both the active and passive LACP modes allow interfaces to negotiate with partner interfaces to determine if they can form an EtherChannel based on criteria such as interface speed and, for Layer 2 EtherChannels, trunking state, and VLAN numbers.

We’ll be configuring our EtherChannel for LACP so we’ll use channel-group x mode active on both the Cisco 3750 and 2950 switches.

Sample Topology

Cisco Catalyst 3750E Switch

enable
config t
interface gig1/0/37
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active
interface gig1/0/38
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active

interface gig1/0/47
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 4 mode active
interface gig1/0/48
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 4 mode active
exit
exit

Cisco Catalyst 2950 Switch 1 & 2

enable
config t
vlan 100
name "192-168-100-0/24"
exit
vlan 200
name "192-168-200-0/24"
exit

interface vlan 100
ip address 192.168.100.40 255.255.255.0
no shut
exit

spanning-tree mode mst

spanning-tree mst configuration
 name AcmeNetworks
 revision 1
 instance 1 vlan 100
 instance 2 vlan 200

interface fas0/1
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface fas0/2
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface port-channel 1
switchport mode trunk

interface fas0/31
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

interface fas0/32
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

interface port-channel 2
switchport mode trunk
exit
exit

That’s all well and good but I’m sure you want to see the output… is it working as expected?

Cisco Catalyst 2950 Switch #1

C2950-SW1#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0064.xxxx.4d80   7s    0x3     0x126    0x3D
Fa0/2     SA      32768     0064.xxxx.4d80  21s    0x3     0x127    0x3D

Channel group 2 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/31    SA      32768     0019.xxxx.49c0  23s    0x2     0x1F     0x3D
Fa0/32    SA      32768     0019.xxxx.49c0  21s    0x2     0x20     0x3D

C2950-SW1#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    16384
             Address     3475.xxxx.a400
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0018.xxxx.4a40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Desg FWD 100000    128.66   P2p

MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    16385
             Address     54e0.xxxx.d441
             Cost        110000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0018.xxxx.4a40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Desg FWD 100000    128.66   P2p

MST02
  Spanning tree enabled protocol mstp
  Root ID    Priority    16386
             Address     0064.xxxx.4d80
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0018.xxxx.4a40
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Desg FWD 100000    128.66   P2p

Let’s have a look at the other Cisco 2950 switch;

Cisco Catalyst 2950 Switch #2

C2950-SW2#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0064.xxxx.4d80   9s    0x4     0x130    0x3D
Fa0/2     SA      32768     0064.xxxx.4d80  10s    0x4     0x131    0x3D

Channel group 2 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/31    SA      32768     0018.xxxx.4a40   4s    0x2     0x1F     0x3D
Fa0/32    SA      32768     0018.xxxx.4a40  25s    0x2     0x20     0x3D

C2950-SW2#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    16384
             Address     3475.xxxx.a400
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0019.xxxx.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p

MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    16385
             Address     54e0.xxxx.d441
             Cost        110000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0019.xxxx.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p

MST02
  Spanning tree enabled protocol mstp
  Root ID    Priority    16386
             Address     0064.xxxx.4d80
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0019.xxxx.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p

We can see that ports Fas0/31 and Fas0/32 that make up Port-Channel 2 are in Alternate Blocking mode on SW2. This is expected since the bridge address of SW1 (0018.xxxx.4a40) is lower than SW2 (0019.xxxx.49c0) and the links are equal cost.

Cheers!
Image credit: Ravenel Bridge Charleston by Roger Kirby

Let’s look at expanding the topology from our last post adding a Cisco 3750E;

Again, that’s pretty straight forward and isn’t too exciting. Although if we leave every uplink/downlink as a member of VLAN 100 and VLAN 200 we’ll end up with a loop in our topology – not a Spanning Tree Loop. What if we add Multiple Spanning Tree Protocol (MSTP) to our configuration just to make it interesting? Our topology might look like this with 2 instances of MSTP running, one for each VLAN.

We’ll make the Avaya switch the root bridge for CIST. We’ll make the Juniper switch the root bridge for MST 1, and we’ll make the Cisco switch the root bridge for MST 2.

That’s interesting… let’s see what we need to-do in order to configure everything up. I’m going to pickup the configuration as I had it setup in the previous post, LACP Configuration  Examples (Part 4). We’ll need to add another LACP group/pair to our Avaya and Juniper switches as well as configure the Cisco switch. We’ll also need to enable MSTP on each switch, add the VLANs to the correct MSTP instances and set the correct bridge priority for each.

Juniper EX2200-C Switch

configure
set chassis aggregated-devices ethernet device-count 2

delete interfaces ge-0/0/4 unit 0
delete interfaces ge-0/0/5 unit 0

set interfaces ge-0/0/4 ether-options 802.3ad ae1
set interfaces ge-0/0/5 ether-options 802.3ad ae1
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp periodic fast

set interfaces ae1 unit 0 family ethernet-switching
set interfaces ae1 unit 0 family ethernet-switching port-mode trunk
set interfaces ae1 unit 0 family ethernet-switching port-mode trunk vlan members VLAN-100 members VLAN-200

delete protocols rstp

set protocols mstp configuration-name AcmeNetworks
set protocols mstp revision-level 1
set protocols mstp msti 1 vlan 100
set protocols mstp msti 2 vlan 200

set protocols mstp msti 1 bridge-priority 16384
commit and-quit

Avaya Ethernet Routing Switch 5520

config t
spanning-tree mode mst
exit
boot

You’ll need to reboot the switch in order to enable MSTP, so go ahead and reboot before continuing the steps;

config t
vlan ports 25,26 tagging tagAll

interface fastEthernet 25,26
lacp key 25
lacp mode active
lacp timeout-time short
lacp aggregation enable
exit

spanning-tree mstp msti 1
spanning-tree mstp msti 1 add-vlan 100
spanning-tree mstp msti 2
spanning-tree mstp msti 2 add-vlan 200
spanning-tree mstp priority 4000

You’ll notice that the Avaya switch accepts a hexadecimal value for the priority, so 4000 in hex = 16384 in decimal.

spanning-tree mstp region region-name AcmeNetworks
spanning-tree mstp region region-version 1
exit

Cisco Catalyst 3750E Switch

config t
vlan 100
name "192-168-100-0/24"
exit
vlan 200
name "192-168-200-0/24"
exit

interface vlan 100
ip address 192.168.100.30 255.255.255.0
no shut
exit

interface vlan 200
ip address 192.168.200.30 255.255.255.0
no shut
exit

interface gig1/0/13
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface gig1/0/14
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active

interface gig1/0/25
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

interface gig1/0/26
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 2 mode active

spanning-tree mode mst

spanning-tree mst configuration
name AcmeNetworks
revision 1
instance 1 vlan 100
instance 2 vlan 200
exit
spanning-tree mst 2 priority 16384
exit

Let’s have a look at our work and see what everything looks like from both a LACP and Spanning Tree perspective.

Cisco Catalyst 3750E Switch

Switch#show lacp neighbor
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/13 FA 127 54e0.xxxx.d440 5s 0x0 0x2 0x3 0x3F
Gi1/0/14 FA 127 54e0.xxxx.d440 5s 0x0 0x2 0x4 0x3F

Channel group 2 neighbors

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/25 FA 32768 3475.xxxx.a400 14s 0x0 0x3019 0x19 0x3F
Gi1/0/26 FA 32768 3475.xxxx.a400 16s 0x0 0x3019 0x1A 0x3F

Switch#show spanning-tree

MST0
Spanning tree enabled protocol mstp
Root ID Priority 16384
Address 3475.xxxx.a400
Cost 0
Port 496 (Port-channel2)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address 0064.xxxx.4d80
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 10000 128.488 P2p
Po2 Root FWD 10000 128.496 P2p

MST1
Spanning tree enabled protocol mstp
Root ID Priority 16385
Address 54e0.322a.d441
Cost 10000
Port 488 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 0064.xxxx.4d80
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 10000 128.488 P2p
Po2 Desg FWD 10000 128.496 P2p

MST2
Spanning tree enabled protocol mstp
Root ID Priority 16386
Address 0064.xxxx.4d80
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 16386 (priority 16384 sys-id-ext 2)
Address 0064.xxxx.4d80
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 10000 128.488 P2p
Po2 Desg FWD 10000 128.496 P2p

We can see that LACP is up and running to both the Avaya and Juniper switches. We can also see that the Cisco switch is the root bridge for MSTI 2 and the root port for MSTI 1 is Port-channel 1 (link to Juniper EX2200-C) while the root port for the CIST is Port-channel2 (link to Avaya ERS 5520). All ports are designated and forwarding traffic.

 Juniper EX2200-C Switch

root> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Fast Active
ge-0/0/1 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/1 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Fast periodic Collecting distributing
ge-0/0/1 Current Fast periodic Collecting distributing

Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/4 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/4 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/5 Actor No No Yes Yes Yes Yes Fast Active
ge-0/0/5 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/4 Current Slow periodic Collecting distributing
ge-0/0/5 Current Slow periodic Collecting distributing

root> show spanning-tree bridge

STP bridge parameters
Context ID : 0
Enabled protocol : MSTP

STP bridge parameters for CIST
Root ID : 16384.34:75:xx:xx:a4:00
Root cost : 0
Root port : ae0.0
CIST regional root : 16384.34:75:xx:xx:a4:00
CIST internal root cost : 10000
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Hop count : 19
Message age : 0
Number of topology changes : 2
Time since last topology change : 14690 seconds
Topology change initiator : ae0.0
Topology change last recvd. from : 34:75:xx:xx:a4:01
Local parameters
Bridge ID : 32768.54:e0:xx:xx:d4:41
Extended system ID : 0
Internal instance ID : 0

STP bridge parameters for MSTI 1
MSTI regional root : 16385.54:e0:xx:xx:d4:41
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Number of topology changes : 5
Topology change initiator : ae1.0
Topology change last recvd. from : 00:64:xx:xx:4d:8d
Local parameters
Bridge ID : 16385.54:e0:xx:xx:d4:41
Extended system ID : 0
Internal instance ID : 1

STP bridge parameters for MSTI 2
MSTI regional root : 16386.00:64:xx:xx:4d:80
Root cost : 10000
Root port : ae1.0
Hello time : 2 seconds
Maximum age : 20 seconds
Forward delay : 15 seconds
Hop count : 19
Number of topology changes : 6
Topology change initiator : ae1.0
Topology change last recvd. from : 00:64:xx:xx:4d:8d
Local parameters
Bridge ID : 32770.54:e0:xx:xx:d4:41
Extended system ID : 0
Internal instance ID : 2

Avaya Ethernet Routing Switch 5520

5520-48T-PWR#show lacp port 13-14,25-26
Admin Oper Trunk Partner
Port Priority Lacp A/I Timeout Key Key AggrId Id Port Status
---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
13 32768 Active A Short 1 12289 8224 32 1 Active
14 32768 Active A Short 1 12289 8224 32 2 Active
25 32768 Active A Short 25 12313 8223 31 282 Active
26 32768 Active A Short 25 12313 8223 31 283 Active

5520-48T-PWR#show spanning-tree mstp config
Maximum Mst Instance Number: 8
Number of Msti Supported: 2
Cist Bridge Priority (hex): 4000
Stp Version: Mstp Mode
Cist Bridge Max Age: 20 seconds
Cist Bridge Forward Delay: 15 seconds
Tx Hold Count: 3
Path Cost Default Type: 32-bit
Max Hop Count: 2000

VLAN members
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
1

Msti Config Id Selector: 0
Msti Region Name: AcmeNetworks
Msti Region Version: 1
Msti Config Digest: 6D:A4:B5:0C:4F:D5:87:75:7E:EF:03:56:75:36:05:E1

5520-48T-PWR#show spanning-tree mstp msti config 1
Msti Bridge Regional Root:  40:00:54:E0:xx:xx:D4:41
Msti Bridge Priority (hex): F000
Msti Root Cost:             10000
Msti Root Port:             MLT 32
Msti State:                 Enabled

VLAN members
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
100

5520-48T-PWR#show spanning-tree mstp msti config 2
Msti Bridge Regional Root:  40:00:00:64:xx:xx:4D:80
Msti Bridge Priority (hex): F000
Msti Root Cost:             10000
Msti Root Port:             MLT 31
Msti State:                 Enabled

VLAN members
------ ------ ------ ------ ------ ------ ------ ------ ------ ------
200

5520-48T-PWR#show spanning-tree mstp msti port role 1
Port Role State STP Status Oper Status
---- ---------- ---------- ---------- -----------
13 Root Forwarding Enabled Enabled
14 Root Forwarding Enabled Enabled
25 Alternate Discarding  Enabled Enabled
26 Alternate Discarding  Enabled Enabled

5520-48T-PWR#show spanning-tree mstp msti port role 2
Port Role State STP Status Oper Status
---- ---------- ---------- ---------- -----------
13 Alternate Discarding  Enabled Enabled
14 Alternate Discarding  Enabled Enabled
25 Root Forwarding Enabled Enabled
26 Root Forwarding Enabled Enabled

We can see from the output above that ports 13,14 are Alternate Discarding for MSTI 1 while ports 25,26 are Alternate Discarding for MSTI 2.

In the output we can see which port is the root bridge port for each switch, we can also see the MSTP config digest which should match on every switch in the topology. In order for the configuration to be valid the MST region name, version and config selector need to match along with correct VLAN IDs matched to the correct MST instance.

Cheers!
Image Credit: New York City Brooklyn Bridge by Diogo Ferrari

In June we started deploying our secondary data center with the intent of providing our own business continuity and disaster recovery services for our tier 1 applications including all our data storage needs. The design allows us the flexibility to utilize both DCs in an active/active configuration with the ability to move workloads (virtual machines) between DCs. While the design allows us that option we’re still testing how we’re going to handle all the different disaster scenarios – blade, enclosure, rack, SAN, cage, entire data center, etc. While our primary data center rings in at 800 sq ft our secondary data center is only 300 sq ft. This is possible because we’re utilizing a traditional disaster recovery model for our big box non-tier 1 applications that for one reason or another aren’t virtualized. This helps reduce the number of lazy assets hanging around and helps control some of the budget numbers. I totally expect the number of big box applications to continue to shrink over time as more and more application vendors embrace virtualization.

We’ve had pretty good success with the design of our first data center so we only made a few corrections. There’s a lot of logistics that need to be considered in any design especially around all the power and cooling requirements.

The Equipment

What equipment did we use? We already deployed Cisco at our primary data center so we decided to stay with Cisco at our secondary data center.

• Cisco Nexus 7010
• Cisco Nexus 5010
• Cisco Nexus 2248
• Cisco Nexus 1000V
• Cisco Catalyst 3750X
• Cisco Catalyst 2960G
• Cisco ASA5520
• Cisco ACE 4710
• Cisco 3945 Router (Internet)
• Cisco 2811 Router (internal T1 locations)

What racks did we use for the network equipment?

• Liebert Knurr Racks
• Liebert MPH/MPX PDUs

What equipment did we use for the servers/blades?

• HP Rack 10000 G2
• HP Rack PDU (AF503A)
• HP IP KVM Console (AF601A)
• HP BladeSystem C7000 Enclosure
• HP Virtual Connect Flex-10 Interconnect
• HP SAN 8Gb Interconnect
• Cisco Catalyst 3120X
• HP BL460c G7
• HP BL620c G7
• HP DL380 G8
• HP DL360 G8

What are we using for storage?

• IBM XIV System Storage Gen3 (SAN) (w/4 1Gbps iSCSI replication ports)
• IBM SAN80B-4 SAN Switch
• EMC DD860 (Disk-Disk backup via Symantec NetBackup)

Additional miscellaneous equipment;

• MRV LX-4048T (terminal server)

We had some challenges with designing our secondary data center due to the density of our equipment. We had to stay under the maximum kw per sq foot load that the room (data center) was designed to handle. This is a simple calculation based on the kW utilization of the equipment to determine if there is adequate power and cooling available to meet that demand. We also had to maintain a N+1 design so we really can’t consuming more than 40% of our capacity leaving 10% for reserve. While some vendors charge a flat fee for the space (includes power) others charge per kWh so it’s very important to understand what type of demand you’re going to be placing on the data center.

My Design

We stood up a pair of Ciena 5200s from Zayo (formerly AboveNet) providing us a DWDM ring with 4 wavelengths between our primary data center and secondary data center . We’re using 2 wavelengths for the IP network between 2 pairs of Cisco Nexus 7010s and 2 wavelengths for the SAN fiber channel network between 2 pair of IBM SAN switches. We have the option of adding upwards of 4 additional wavelengths before we need to add any hardware so we have room for growth. The 4 wavelengths are diverse between an east and west path but they are not protected so it’s up to the higher layer protocols to provide the redundancy and failover.Not visible in the diagram above is a 10GE WAN ring that connects all our hospitals together. The primary and secondary data centers are also tied into that ring via multiple peering points for redundancy. You might be asking yourself why I’m using a Cisco 3750E as a termination switch in our primary data center. At the time we deployed our Cisco Nexus 7010s they didn’t support the 10GBase-ER SFP+ optic so I had to use the Cisco 3750E (with RPSU) as a glorified media transceiver/converter from 10GBase-ER to 10GBase-SR. The Cisco Nexus 7010 now has a 10GBase-ER SFP+ optic available so we didn’t need to use the Cisco 3750 in the secondary data center.

We are essentially stretching a Layer 2 vPC connection between the 2 data centers. It’s possible that some folks will get excited at the mention of Layer 2 between the data centers but it’s the best solution for us at this time and it certainly has pros and cons like everything in networking. We looked at potentially running OTV between the Cisco Nexus 7010s but ultimately decided to use a vPC configuration. We are only stretching the virtual machine VLANs that we need between the data centers.

My Thoughts

There’s a lot of work required to design any data center or even an ICR (Intermediate Communications Room), CCR (Central Communications Room), MDF (Main Distribution Frame) or IDF (Intermediate Distribution Frame). You’re immediately confronted with space, power and cooling challenges never mind coming up with the actual IP addressing scheme, VLAN assignments, routing vs bridging ,etc. You need to determine how much cabling you’ll need both CAT6 and fiber, perhaps you’ll look to use twinax of DAC (Direct Attach Copper) for your 10GE connections. Let’s not forget to include the ladder racks, basket trays, fiber conduits, PDUs, out-of-band networking, etc.

You also need to design the data center as if it was 300+ miles away… license those iLOs (HP Integrated Lights Out), purchase IP enabled KVMs, purchase console/terminal servers (Opengear or MRV) and wire everything up as if you will never have the opportunity to visit it again. We’ve had a few issues in the past few years that were quickly (less than 15 minutes) resolved thanks to having all our iLOs licensed, all our KVMs IP enabled, all our console/serial ports connected to a console/terminal server and the ability to dial-up into the console/terminal server should the problem get really bad.

Here’s a short story… We had a number of billing issues in the first few months of our contract with our current primary data center provider and the data from our Liebert PDUs, HP PDUs, and HP C7000 enclosures was invaluable in calling into question the numbers that were being reported to us. In all honesty when they told me we were consuming 53A on a 50A circuit I knew that something was grossly wrong with their math. In the end the provider admitted that there numbers were grossly wrong and the corrected numbers were in-line with the data we collected from our equipment.

It’s never a good idea to skimp on the documentation and I really advise taking lots of pictures, you’d be surprised how quickly you can forget what the back a specific rack looks like when you’re trying to walk Smart Hands through replacing a component at 2AM in the morning.

Cheers!

With previous Cisco switches such as the 6509, 3750, 2960, etc we know that the following commands (when sent via a Perl script using the Net-SNMP Perl module) would instruct the switch to copy it’s running-config to a TFTP server.

snmpset -v1 -c$COMMUNITY $HOST ccCopyProtocol.$RANDOM i 1
snmpset -v1 -c$COMMUNITY $HOST ccCopySourceFileType.$RANDOM i 4
snmpset -v1 -c$COMMUNITY $HOST ccCopyDestFileType.$RANDOM i 1
snmpset -v1 -c$COMMUNITY $HOST ccCopyServerAddress.$RANDOM a "10.1.1.50"
snmpset -v1 -c$COMMUNITY $HOST ccCopyFileName.$RANDOM s "sw-train-acme.cfg"
snmpset -v1 -c$COMMUNITY $HOST ccCopyEntryRowStatus.$RANDOM i 1
sleep 5
snmpget -v1 -c$COMMUNITY $HOST ccCopyState.$RANDOM
#if not successful sleep 3 and re-check ccCopyState else continue and destroy table entry
snmpset -v1 -c$COMMUNITY $HOST ccCopyEntryRowStatus.$RANDOM i 6

I know that the both the Cisco Nexus 7010 and 5010 both balk at the SNMP OIDS/MIBS used above. So I’m searching for a set of equivalent SNMP OIDS/MIBS as those in CISCO-CONFIG-COPY-MIB for NX-OS. I’m not sure that such a OID/MIB even exists for NX-OS but it doesn’t hurt to search and ask.

I’m curious if anyone else has come across this issue? I know that there is an XML interface available but I would prefer to keep using the PERL/SNMP script that I’ve already developed. In the interim I’ll probably write an Expect script (or add some Expect code to my existing Perl script) to remotely connect to the switches and issue the appropriate copy commands.

Cheers!

Updated: Monday June 27, 2011

I’ve finally found the issue and now I’m able to backup the Cisco Nexus switches as expected.
[ad name=”ad-articlefooter”]

It’s definitely well worth the time to review.

Cheers!

It’s definitely well worth the time to review.

Cheers!

I’ve essentially boiled my options down to two possible solutions (vendors);

Juniper SRX 240

Cisco ASA 5550

So which do I choose and how to best evaluate the different products. The primary purpose of the device is to provide branch office IPSec tunnels. The product needs to support OSPF and it needs some limited support for Multicast over VPN.

This morning I was lucky enough to have one of our preferred vendors, who just happens to be a Juniper reseller, come on site and help setup 2 Juniper SRX 210 gateways for us to demo.  I’ve never worked with a Junos based product and while the web based GUI was fairly straightforward the CLI interface is going to take some time to get use to. It’s not like Cisco, or Nortel or Brocade, or Blade Technologies. Thankfully I did find a quick start guide that helped get my feet wet with Junos.

Once I’m done with the Juniper SRX I’ll need to turn my attention to the Cisco ASA (Tom you know what I’ll be calling for soon – demo time).

I’ll post a summary once I have some thoughts about the Juniper SRX. Anyone care to comment regarding either the Juniper SRX or the Cisco ASA as it pertains to branch office VPN tunnels? As a note I’m already migrating our Nortel VPN end-users to our Juniper SSL VPN Secure Access 4000 appliances.

Cheers!

I applaud Nortel for for making these Technical Configuration Guides available to users. It helps to broaden Nortel’s product reach and it empowers Nortel users to understand the different configuration options. I’d like to see Nortel get these documents indexed by Google or even there own internal Knowledgebase search engine.

In short you can use LLDP (802.1ab) on a Cisco 3750  to configure the Voice VLAN much the same way as you might on a Nortel ERS5520 switch using ADAC/LLDP. For those Cisco shops that are using Nortel Succession for voice this should be a welcome capability.

As in the past I’m going to post the document on my website;

http://www.michaelfmcnamara.com/files/ClientsInteropCisco_L2_1.1.pdf

As I’ve commented in the past it’s quite possible that Nortel may object to my “copying” of these documents. The goal is to make them readily available to Nortel users and allow them to be indexed by Google. I believe there are now quite a few Nortel corporate folks and engineers reading this blog, please contact me if you have concerns about me posting these documents.

Cheers!