Michael McNamara

How does latency impact network throughput?

Michael McNamara — Tue, 28 Sep 2021 16:49:15 +0000

I was recently having a conversation with a DevOps colleague (let’s not jeer too loudly) who was trying to understand why he wasn’t getting more than 350Mbps between two servers over a 1Gbps WAN connection. He thought there must be a problem with the network and suggested that I should open a ticket with the carrier to “fix” the issue. I attempted to explain to him that it was the latency and distance between the two servers (3,000 miles) that was limiting the TCP performance and he could potentially overcome that issue by using multiple TCP sockets with larger TCP window sizes, or potentially switch to UDP instead of TCP.

I used iPerf3 to demonstrate the issue… with a single stream/thread we were able to achieve ~ 350Mbps. With a second stream/thread we were able to hit ~ 600Mbps. With a third stream/thread we were able to hit ~ 789Mbps.

It wasn’t magic…. it’s the well known fact that latency plays a huge role in TCP performance. In order to understand why it impacts TCP performance you need to understand how TCP works. TCP requires that transmitted data sets are acknowledged before the next set of data can be transmitted. The TCP window size determines the size of those data sets, larger TCP window size allows more data to be transmitted before an acknowledgement is required. The delay in getting the acknowledgement back is what limits the performance.

There is a well written blog article from Netbeez written by Stefano Gridelli titled, Impact of Packet Loss and Round-Trip Time on Throughput that covers this topic in great detail. You can even apply a mathematical formula to determine the max potential throughput given a known RTT latency.

Cheers!

Airconsole – Portable Serial Ethernet/WiFi/Bluetooth Console Server

Michael McNamara — Mon, 06 Mar 2017 22:48:48 +0000

Over the past Christmas holiday I received an Airconsole XL 2.o from a good friend and industry colleague. I’m a big proponent of having a serial out-of-band network for all my data centers. It’s a relatively cheap insurance policy designed to prevent outages from being any longer than they absolutely need to be. There’s nothing worse than waiting for “smart hands” to get a laptop plugged into your core Cisco Nexus 7010 when your network is down hard. While I’m a big fan of Opengear, there’s definitely a niche for a small portable wireless device that you quickly and easily deploy to provide access to a serial/console port.

Image from www.get-console.com

Getting Started

I’ll freely admit that I tried to employ Airconsole before reading the manual and initially failed. After coming back to the documentation I was able to figure out the following pieces of information;

The toggle switch is for O)ff C)harge and R)un
The default WiFi SSID is AirConsole-AC with a password of 12345678
The default password for the management page http://192.168.10.1 is admin/admin
The default TCP port to establish a telnet connection to the serial port is 3696.
The default TCP port to establish a SSH connection to the serial port is 4001 (requires firmware 2.77 or later).

After charging the Airconsole XL for the better part of 4 hours (I was busy breaking other things), I decided to give the little device a go. After checking the website documentation I was quickly able to connect to the SSID and using PuTTY establish an telnet session to TCP port 3696 which got me connected to the Juniper EX2200-C that I had connected the Airconsole to.

Firmware Upgrade

There was no option for SSH since the device I received was only running firmware 2.50, so an upgrade was required (SSH is only available in firmware 2.77 and later). I upgrade to firmware 2.80 by following the instructions. I would suggest reviewing the release notes if interested.

With the upgrade completed I had a number of additional options and features including SSH and HTTPS. I was able to establish an SSH session via TCP/4001 using PuTTY. Using both the WiFi and Ethernet connections were pretty straight forward, you can also configure the Airconsole to act as a wireless client – not just an AP.

Bluetooth / Samsung Galaxy Tablet

I was quickly able to pair my Samsung Galaxy Note 10.1 to the Airconsole and using SerialBot app I was quickly able to establish a connection to the console port of my Juniper EX2200-C from my Samsung tablet via Bluetooth.

Now if i could only remember the root password for the Juniper EX2200-C switch ?&!#@^

My Thoughts?

All in all I think the Airconsole XL 2.0 is very useful piece of kit for any network engineer and priced at only $139 its not going to break the bank. While a simple USB to serial cable could also do the job the ease of using an Airconsole will likely pay for itself quickly in making what can sometimes be a time consuming process hassle-free.

How many times have you had to balance your laptop on one hand while typing with the other hand?

Cheers!

LACP Configuration Examples (Part 7)

Michael McNamara — Mon, 06 Jun 2016 22:55:53 +0000

Over the past few weeks I’ve been working with HP switches so I decided I would extend my series on LACP trunking to include HP switches. In my lab I used HP 2810 switches which are dated but the concepts are the same for any of the newer HPE switch equipment. I cabled the HP switches to a pair of Cisco 2950s, you may noticed that I’ve changed some of the ports I’m using from the previous lab examples (check the diagram).

I noticed while working on setting up this lab that the MST digest between the Cisco and HP switches didn’t match. After some quick research it appears that the Cisco 2950s I have in the lab operate with a pre standard MST operation. Other Cisco switches identify them as such and are interoperable but you may have issues with third-party devices that are expecting the 802.1s standard. You can see both digests from the Cisco 3750 below and they match both the Cisco 2950 and the HP 2810 switches.

C3750-SW1#show spanning-tree mst configuration digest
Name      [AcmeNetworks]
Revision  1     Instances configured 3
Digest          0x6DA4B50C4FD587757EEF0356753605E1
Pre-std Digest  0x421D7D23BF9562A0C35E46CA1BE8A75C

Example Topology

You’ll notice the HP switches at the bottom of the diagram. It was pretty straight forward but here’s what I needed to do.

Cisco Catalyst 2950 Switch 1 & 2

First we needed to configure the ports on the Cisco 2950s that would be connected to the HP switches. I used Port Channel 3 for this and enabled LACP;

interface fas0/15
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active

interface fas0/16
switchport mode trunk
channel-protocol lacp
channel-group 3 mode active

HP 2810 Switch 1 & 2

Now we need to configure the HP switches, VLANs, IP addressing, ports, trunking, MST, etc;

vlan 100
name "192-168-100-0/24"
vlan 200
name "192-168-200-0/24"

vlan 100
ip address 192.168.100.70 255.255.255.0
exit

spanning-tree
spanning-tree config-name "AcmeNetworks"
spanning-tree config-revision 1
spanning-tree instance 1 vlan 100
spanning-tree instance 2 vlan 200

trunk 1,13 trk1 lacp
trunk 23,24 trk2 lacp

vlan 100 tagged trk1
vlan 200 tagged trk1

vlan 100 tagged trk2
vlan 200 tagged trk2

That’s all well and good but I’m sure you want to see the output… is it working as expected? Well let’s check it out.

Cisco Catalyst 2950 Switch 1

We can see from the data below that LACP has established to the HP switch and Spanning Tree is working as expected;

C2950-SW1#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0064.40cf.4d80  24s    0x3     0x102    0x3D
Fa0/2     SA      32768     0064.40cf.4d80  17s    0x3     0x103    0x3D

Channel group 2 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/31    SA      32768     0018.ba8e.4a40  22s    0x2     0x1F     0x3D
Fa0/33    SA      32768     0018.ba8e.4a40   3s    0x2     0x21     0x3D

Channel group 3 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/15    SA      0         0026.f1df.f400  21s    0x32    0x18     0x3D
Fa0/16    SA      0         0026.f1df.f400  21s    0x32    0x17     0x3D

C2950-SW1#show spanning-tree

MST00
  Spanning tree enabled protocol mstp
  Root ID    Priority    16384
             Address     3475.c732.a400
             Cost        0
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32768  (priority 32768 sys-id-ext 0)
             Address     0019.2faa.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p
Po3              Desg FWD 100000    128.67   P2p Bound(RSTP)


MST01
  Spanning tree enabled protocol mstp
  Root ID    Priority    16385
             Address     54e0.322a.d441
             Cost        120000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0019.2faa.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p
Po3              Boun FWD 100000    128.67   P2p Bound(RSTP)


MST02
  Spanning tree enabled protocol mstp
  Root ID    Priority    16386
             Address     0064.40cf.4d80
             Cost        100000
             Port        65 (Port-channel1)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32770  (priority 32768 sys-id-ext 2)
             Address     0019.2faa.49c0
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 100000    128.65   P2p
Po2              Altn BLK 100000    128.66   P2p
Po3              Boun FWD 100000    128.67   P2p Bound(RSTP)

C2950-SW1#show spanning-tree mst configuration digest
Name      [AcmeNetworks]
Revision  1
Instance  Vlans mapped
--------  ---------------------------------------------------------------------
0         1-99,101-199,201-4094
1         100
2         200
-------------------------------------------------------------------------------
Digest    421D7D23BF9562A0C35E46CA1BE8A75C

Cisco Catalyst 2950 Switch 2

We can see from the data below that LACP has established to the HP switch and Spanning Tree is working as expected;

C2950-SW2#show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
Channel group 1 neighbors

Partner's information:

LACP port Oper Port Port
Port Flags Priority Dev ID Age Key Number State
Fa0/47 SA 32768 0064.40cf.4d80 2s 0x4 0x130 0x3D
Fa0/48 SA 32768 0064.40cf.4d80 25s 0x4 0x131 0x3D

Channel group 2 neighbors

Partner's information:

LACP port Oper Port Port
Port Flags Priority Dev ID Age Key Number State
Fa0/31 SA 32768 0019.2faa.49c0 27s 0x2 0x1F 0x3D
Fa0/33 SA 32768 0019.2faa.49c0 19s 0x2 0x21 0x3D

Channel group 3 neighbors

Partner's information:

LACP port Oper Port Port
Port Flags Priority Dev ID Age Key Number State
Fa0/15 SA 0 0026.f1e1.41a0 29s 0x32 0x17 0x3D
Fa0/16 SA 0 0026.f1e1.41a0 0s 0x32 0x18 0x3D

C2950-SW2#show spanning-tree

MST00
Spanning tree enabled protocol mstp
Root ID Priority 16384
Address 3475.c732.a400
Cost 0
Port 65 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address 0018.ba8e.4a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100000 128.65 P2p
Po2 Desg FWD 100000 128.66 P2p
Po3 Desg FWD 100000 128.67 P2p Bound(RSTP)

MST01
Spanning tree enabled protocol mstp
Root ID Priority 16385
Address 54e0.322a.d441
Cost 120000
Port 65 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 0018.ba8e.4a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100000 128.65 P2p
Po2 Desg FWD 100000 128.66 P2p
Po3 Boun FWD 100000 128.67 P2p Bound(RSTP)

MST02
Spanning tree enabled protocol mstp
Root ID Priority 16386
Address 0064.40cf.4d80
Cost 100000
Port 65 (Port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32770 (priority 32768 sys-id-ext 2)
Address 0018.ba8e.4a40
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100000 128.65 P2p
Po2 Desg FWD 100000 128.66 P2p
Po3 Boun FWD 100000 128.67 P2p Bound(RSTP)

C2950-SW2# show spanning-tree mst configuration digest
Name [AcmeNetworks]
Revision 1
Instance Vlans mapped
-------- ---------------------------------------------------------------------
0 1-99,101-199,201-4094
1 100
2 200
-------------------------------------------------------------------------------
Digest 421D7D23BF9562A0C35E46CA1BE8A75C

HP 2810 Switch 1

HP-SW1# show lacp

                           LACP

   PORT   LACP      TRUNK     PORT      LACP      LACP
   NUMB   ENABLED   GROUP     STATUS    PARTNER   STATUS
   ----   -------   -------   -------   -------   -------
   1      Active    Trk1      Up        Yes       Success
   13     Active    Trk1      Up        Yes       Success
   23     Active    Trk2      Up        Yes       Success
   24     Active    Trk2      Up        Yes       Success


HP-SW1# show cdp neighbors

 CDP neighbors information

  Port Device ID                     | Platform                     Capability
  ---- ----------------------------- + ---------------------------- -----------
  1    00 26 f1 e1 41 a0             | ProCurve J9021A Switch 28... S
  13   00 26 f1 e1 41 a0             | ProCurve J9021A Switch 28... S
  23   C2950-SW1                     | Cisco Internetwork Operat... S
  24   C2950-SW1                     | Cisco Internetwork Operat... S

HP-SW1# show spanning-tree

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes
  Force Version : MSTP-operation
  IST Mapped VLANs : 1

  Switch MAC Address : 0026f1-dff400
  Switch Priority    : 32768
  Max Age  : 20
  Max Hops : 20
  Forward Delay : 15

  Topology Change Count  : 332
  Time Since Last Change : 53 mins

  CST Root MAC Address : 3475c7-32a400
  CST Root Priority    : 16384
  CST Root Path Cost   : 200000
  CST Root Port        : Trk2

  IST Regional Root MAC Address : 0026f1-dff400
  IST Regional Root Priority    : 32768
  IST Regional Root Path Cost   : 0
  IST Remaining Hops            : 20

  Root Guard Ports :
  TCN Guard Ports  :
  Protected Ports :
  Filtered Ports :

                  |           Prio             | Designated    Hello
  Port  Type      | Cost      rity  State      | Bridge        Time  PtP Edge
  ----- --------- + --------- ----- ---------- + ------------- ----- --- ----
  2     100/1000T | Auto      128   Disabled   |
  3     100/1000T | Auto      128   Disabled   |
  4     100/1000T | Auto      128   Disabled   |
  5     100/1000T | Auto      128   Disabled   |
  6     100/1000T | Auto      128   Disabled   |
  7     100/1000T | Auto      128   Disabled   |
  8     100/1000T | Auto      128   Disabled   |
  9     100/1000T | Auto      128   Disabled   |
  10    100/1000T | Auto      128   Disabled   |
  11    100/1000T | Auto      128   Disabled   |
  12    100/1000T | Auto      128   Disabled   |
  14    100/1000T | Auto      128   Disabled   |
  15    100/1000T | Auto      128   Disabled   |
  16    100/1000T | Auto      128   Disabled   |
  17    100/1000T | Auto      128   Disabled   |
  18    100/1000T | Auto      128   Disabled   |
  19    100/1000T | Auto      128   Disabled   |
  20    100/1000T | Auto      128   Disabled   |
  21    100/1000T | Auto      128   Disabled   |
  22    100/1000T | Auto      128   Disabled   |
  Trk1            | 20000     64    Forwarding | 0026f1-dff400 2     Yes No
  Trk2            | 200000    64    Forwarding | 00192f-aa49c0 2     Yes No

HP-SW1# show spanning-tree instance 1

 MST Instance Information

  Instance ID : 1
  Mapped VLANs : 100

  Switch Priority         : 32768

  Topology Change Count   : 39
  Time Since Last Change  : 53 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 0
  Regional Root Port        : This switch is root
  Remaining Hops            : 20
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Designated Forwarding 0026f1-dff400
  Trk2            200000    128      Master     Forwarding 0026f1-dff400

HP-SW1# show spanning-tree instance 2

 MST Instance Information

  Instance ID : 2
  Mapped VLANs : 200

  Switch Priority         : 32768

  Topology Change Count   : 38
  Time Since Last Change  : 53 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 0
  Regional Root Port        : This switch is root
  Remaining Hops            : 20
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Designated Forwarding 0026f1-dff400
  Trk2            200000    128      Master     Forwarding 0026f1-dff400

HP-SW1# show spanning-tree mst-config

 MST Configuration Identifier Information

  MST Configuration Name : AcmeNetworks
  MST Configuration Revision : 1
  MST Configuration Digest : 0x6DA4B50C4FD587757EEF0356753605E1

  IST Mapped VLANs : 1

  Instance ID Mapped VLANs
  ----------- ---------------------------------------------------------
  1           100
  2           200

HP 2810 Switch 2

HP-SW2# show lacp

                           LACP

   PORT   LACP      TRUNK     PORT      LACP      LACP
   NUMB   ENABLED   GROUP     STATUS    PARTNER   STATUS
   ----   -------   -------   -------   -------   -------
   1      Active    Trk1      Up        Yes       Success
   13     Active    Trk1      Up        Yes       Success
   23     Active    Trk2      Up        Yes       Success
   24     Active    Trk2      Up        Yes       Success


HP-SW2# show cdp neighbors

 CDP neighbors information

  Port Device ID                     | Platform                     Capability
  ---- ----------------------------- + ---------------------------- -----------
  1    00 26 f1 df f4 00             | ProCurve J9021A Switch 28... S
  13   00 26 f1 df f4 00             | ProCurve J9021A Switch 28... S
  23   C2950-SW2                     | Cisco Internetwork Operat... S
  24   C2950-SW2                     | Cisco Internetwork Operat... S

HP-SW2# show spanning-tree

 Multiple Spanning Tree (MST) Information

  STP Enabled   : Yes
  Force Version : MSTP-operation
  IST Mapped VLANs : 1

  Switch MAC Address : 0026f1-e141a0
  Switch Priority    : 32768
  Max Age  : 20
  Max Hops : 20
  Forward Delay : 15

  Topology Change Count  : 65
  Time Since Last Change : 66 mins

  CST Root MAC Address : 3475c7-32a400
  CST Root Priority    : 16384
  CST Root Path Cost   : 200000
  CST Root Port        : Trk1

  IST Regional Root MAC Address : 0026f1-dff400
  IST Regional Root Priority    : 32768
  IST Regional Root Path Cost   : 20000
  IST Remaining Hops            : 19

  Root Guard Ports :
  TCN Guard Ports  :
  Protected Ports :
  Filtered Ports :

                  |           Prio             | Designated    Hello
  Port  Type      | Cost      rity  State      | Bridge        Time  PtP Edge
  ----- --------- + --------- ----- ---------- + ------------- ----- --- ----
  2     100/1000T | Auto      128   Disabled   |
  3     100/1000T | Auto      128   Disabled   |
  4     100/1000T | Auto      128   Disabled   |
  5     100/1000T | Auto      128   Disabled   |
  6     100/1000T | Auto      128   Disabled   |
  7     100/1000T | Auto      128   Disabled   |
  8     100/1000T | Auto      128   Disabled   |
  9     100/1000T | Auto      128   Disabled   |
  10    100/1000T | Auto      128   Disabled   |
  11    100/1000T | Auto      128   Disabled   |
  12    100/1000T | Auto      128   Disabled   |
  14    100/1000T | Auto      128   Disabled   |
  15    100/1000T | Auto      128   Disabled   |
  16    100/1000T | Auto      128   Disabled   |
  17    100/1000T | Auto      128   Disabled   |
  18    100/1000T | Auto      128   Disabled   |
  19    100/1000T | Auto      128   Disabled   |
  20    100/1000T | Auto      128   Disabled   |
  21    100/1000T | Auto      128   Disabled   |
  22    100/1000T | Auto      128   Disabled   |
  Trk1            | 20000     64    Forwarding | 0026f1-dff400 2     Yes No
  Trk2            | 200000    64    Blocking   | 0018ba-8e4a40 2     Yes No

HP-SW2# show spanning-tree instance 1

 MST Instance Information

  Instance ID : 1
  Mapped VLANs : 100

  Switch Priority         : 32768

  Topology Change Count   : 11
  Time Since Last Change  : 66 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 20000
  Regional Root Port        : Trk1
  Remaining Hops            : 19
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Root       Forwarding 0026f1-dff400
  Trk2            200000    128      Alternate  Blocking   0026f1-e141a0

HP-SW2# show spanning-tree instance 2

 MST Instance Information

  Instance ID : 2
  Mapped VLANs : 200

  Switch Priority         : 32768

  Topology Change Count   : 10
  Time Since Last Change  : 66 mins

  Regional Root MAC Address : 0026f1-dff400
  Regional Root Priority    : 32768
  Regional Root Path Cost   : 20000
  Regional Root Port        : Trk1
  Remaining Hops            : 19
                                                           Designated
  Port  Type      Cost      Priority Role       State      Bridge
  ----- --------- --------- -------- ---------- ---------- -------------
  2     100/1000T Auto      128      Disabled   Disabled
  3     100/1000T Auto      128      Disabled   Disabled
  4     100/1000T Auto      128      Disabled   Disabled
  5     100/1000T Auto      128      Disabled   Disabled
  6     100/1000T Auto      128      Disabled   Disabled
  7     100/1000T Auto      128      Disabled   Disabled
  8     100/1000T Auto      128      Disabled   Disabled
  9     100/1000T Auto      128      Disabled   Disabled
  10    100/1000T Auto      128      Disabled   Disabled
  11    100/1000T Auto      128      Disabled   Disabled
  12    100/1000T Auto      128      Disabled   Disabled
  14    100/1000T Auto      128      Disabled   Disabled
  15    100/1000T Auto      128      Disabled   Disabled
  16    100/1000T Auto      128      Disabled   Disabled
  17    100/1000T Auto      128      Disabled   Disabled
  18    100/1000T Auto      128      Disabled   Disabled
  19    100/1000T Auto      128      Disabled   Disabled
  20    100/1000T Auto      128      Disabled   Disabled
  21    100/1000T Auto      128      Disabled   Disabled
  22    100/1000T Auto      128      Disabled   Disabled
  Trk1            20000     128      Root       Forwarding 0026f1-dff400
  Trk2            200000    128      Alternate  Blocking   0026f1-e141a0

HP-SW2#  show spanning-tree mst-config

 MST Configuration Identifier Information

  MST Configuration Name : AcmeNetworks
  MST Configuration Revision : 1
  MST Configuration Digest : 0x6DA4B50C4FD587757EEF0356753605E1

  IST Mapped VLANs : 1

  Instance ID Mapped VLANs
  ----------- ---------------------------------------------------------
  1           100
  2           200

Cheers!

How to enable SSLv3 on Firefox for network management

Michael McNamara — Sun, 22 Feb 2015 21:53:02 +0000

I recently tried to log into a Gigamon GigaVUE-420 that was running an older software release. I quickly found that I was unable to connect to the web management interface because the Gigamon was utilizing an SSLv3 cipher which has been disabled in almost every browser including, Internet Explorer, Chrome and Firefox. I received the descript error, ssl_error_no_cypher_overlap when I tried to connect to the management interface.
Thankfully I found a quick work around in Firefox to allow SSLv3 ciphers which allowed me to continue with my work. If you go to about:config within Firefox you can change the minimum TLS version from 1 to 0.

This will allow Firefox to negotiate an SSLv3 cipher with whatever appliance or management interface until you can upgrade or reconfigure them.

Now I need to due some research to see if the later software releases of the Gigamon have addressed the issue with SSLv3.

Cheers!

Image Credit: Mario

BGP Multihomed Internet Data Center

Michael McNamara — Mon, 15 Dec 2014 13:00:56 +0000

It’s both loved and loathed in the network engineering community but BGP came through for us in the past 24 hours.

We utilize BGP to provide dynamic routing between the many Internet Service Providers we are peering with and at the many Data Centers and circuits over which we peer. This past weekend we had an issue with our primary Internet Service Provider (AT&T) but BGP did it’s job and dutifully detected the dead router and re-routed traffic to the remaining Internet Service Providers. The actual outage time was less then 60 seconds. Even though it occurred around 1:30AM EST in the morning we’re hosting websites that need to be accessible in every timezone around the world. While it was 1:30AM on the East coast it was only 10:30PM on the West coast where shoppers were still busy picking through the online goods and placing orders. And while it might have been a little too early for our friends in the UK (6:30AM GMT), we could have shoppers online from either France or Germany (7:30AM GMT +1).

Dec 14 2014 01:27:20.337: %BGP-5-ADJCHANGE: neighbor 12.251.xxx.xxx Down BGP Notification sent
Dec 14 2014 01:27:20.337: %BGP-3-NOTIFICATION: sent to neighbor 12.251.xxx.xxx 4/0 (hold time expired) 0 bytes
Dec 14 2014 01:27:22.650: %BGP_SESSION-5-ADJCHANGE: neighbor 12.251.xxx.xxx IPv4 Unicast topology base removed from session  BGP Notification sent
Dec 14 2014 01:33:25.052: %BGP-5-ADJCHANGE: neighbor 12.251.xxx.xxx Up

We also utilize BGP internally in combination with BFD (Bidirectional Forwarding Detection) to help reduce the failover time on the internal network. We’ve actually had BFD accidentally trip a number of times because it can be too sensitive which can create just as many issues having routes flapping back and forth between multiple paths.

As of this writing I have ~ 511,000 IP routes in my BGP routing tables.

Looking at a peering point on the East coast of the United States;

511499 network entries using 132989740 bytes of memory
2550193 path entries using 244818528 bytes of memory
836860/82048 BGP path/bestpath attribute entries using 187456640 bytes of memory
293120 BGP AS-PATH entries using 13504896 bytes of memory
12459 BGP community entries using 1489256 bytes of memory
51 BGP route-map cache entries using 3264 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 580262324 total bytes of memory
BGP activity 11331307/10819807 prefixes, 150321731/147771538 paths, scan interval 60 secs

Here’s a look at a peering point on the West coast of the United States;

511029 network entries using 132867540 bytes of memory
1021218 path entries using 98036928 bytes of memory
246998/81716 BGP path/bestpath attribute entries using 55327552 bytes of memory
145392 BGP AS-PATH entries using 6562258 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 292794278 total bytes of memory
BGP activity 4607568/4096503 prefixes, 24245483/23224265 paths, scan interval 60 secs

The delta in path entries between the two is a result of the number of BGP peers I have on that specific router.

East = 2,550,193
West = 1,021,218

As you can guess I have a number of additional peers on the East coast than I have on the West coast – plans are in the works to resolve that next calendar year.

You can see the dramatic growth in the number of BGP routes being advertised over the Internet from http://bgp.potaroo.net/.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 21 of 30. All the posts can be viewed from the 30in30 tag.

Akamai CDN and TCP Connections

Michael McNamara — Fri, 22 Aug 2014 01:27:37 +0000

In my latest adventure I had to untangle the interaction between a pair of Cisco ACE 4710s and Akamai’s Content Distribution Network (CDN) including SiteShield, Mointpoint, and SiteSpect. It’s truly amazing how complex and almost convoluted a CDN can make any website. Any when it fails you can guess who’s going to get the blame. Over the past few weeks I’ve been looking at a very interesting problem where an Internet facing VIP was experiencing a very unbalanced distribution across the real servers in the severfarm. I wrote a few quick and dirty Bash shell scripts to-do some repeated load tests utilizing curl and sure enough I was able to confirm that there was something amiss between the CDN and the LB. If I tested against the origin VIP I had near perfect round-robin load-balancing across the real servers in the VIP, if I tested against the CDN I would get very uneven load-balancing results.

When a web browser opens a connection to a web server it will generally send multiple requests across a single TCP connection similar to the figure below. Occasionally some browsers will even utilize HTTP pipelining if both the server and browser support that feature, sending multiple requests without waiting for the corresponding TCP acknowledgement.

The majority of load balancers, including the Cisco ACE 4710 and the A10 AX ADC/Thunder, will look at the first request in the TCP connection and apply the load-balancing metric and forward the traffic to a specific real server in the VIP. In order to speed the processing of future requests the load balancer will forward all traffic in that connection to the same real server in the VIP. This generally isn’t a problem if there’s only a single user associated with a TCP connection.

Akamai will attempt to optimize the number of TCP connections from their edge servers to your origin web servers by sending multiple requests from different users all over the same TCP connection. In the example below there are requests from three different users but it’s been my experience that you could see requests for dozens or even hundreds of users across the same TCP connection.

And here lies the problem, the load balancer will only evaluate the first request in the TCP connection, all subsequent requests will be sent to the same real server leaving some servers over utilized and others under utilized.

Thankfully there are configuration options in the majority of load balancers to work around this problem and instruct the load balancer to evaluate all requests in the TCP connection independently.

A10 AX ADC/Thunder

strict-transaction-switch

Cisco ACE 4710

parameter-map type http HTTP_PARAMETER_MAP
  persistence-rebalance strict

With the configuration change made now every request in the TCP connection is evaluated and load-balanced independently resulting in a more even distribution across the real servers in the farm.

In this scenario I’m using HTTP cookies to provide session persistence and ‘stickiness’ for the user sessions. If your application is stateless then you don’t really need to worry that a user lands on the same real server for each and every request.

Cheers!

Image Credit: topfer

Application Packet Loss and Performance

Michael McNamara — Fri, 23 May 2014 13:56:04 +0000

I had an interesting problem this past week where a vendor tried to tell me that “no other customers were having issues“. How many times have you heard that line? The problem started with the application folks coming over to ask if there were any network issues. In a short discussion with them I learned that they had application interfaces that were taking upwards of 20-40 seconds to complete a transaction exchange and that was causing their transaction queues to back up and fall behind.

It’s fascinating now that I’m working in retail to follow the actual process flow from order entry to order fulfillment.

In any case a few quick tests using ICMP pings didn’t show any issues or problems. However, a subsequent packet trace performed from the server revealed a large number of TCP Re-transmissions and Duplicate ACKs. It was pretty clear to me that we had some significant packet loss between the two servers. However the vendor felt it was indicative of “application packet loss“. I’ve been in the networking field for quite a few years now… I’ve seen a lot and heard a lot but I’ve never heard the phrase “application packet loss”. The vendor was suggesting that it was the application that was causing the TCP Re-transmissions and Duplicate ACKs and that the network was not to blame.

In classic fashion I politely called bullshit, ok maybe I was a little more forceful than that.

It was the TCP Re-transmissions that was causing the slow down in the transaction exchanges. The packets were being re-transmitted because they were being lost somewhere between the two servers. I could see a 8 second delay here, a 8 second delay there… when you add them up you get interfaces that generally take 200ms to exchange data taking upwards of 20-40 seconds. The larger problem, we had 5000+ transactions backing up and we were falling further and further behind since the rate at which the transactions were entering the queue was far outpacing the rate at which the transactions were being processed.

In the end the vendor changed some of their Internet BGP peering in order to leverage a different Internet provider and path and that magically solved the problem instantly. There was some peering point out on the Internet that was throwing out some packets and that was causing our issues.

If you’ve ever heard of application packet loss please by all means please educate me!

Cheers!

References;
Why does packet loss destroy application performance over the WAN? by Andy Gottlieb

Secondary Data Center – Where have I been?

Michael McNamara — Sat, 08 Dec 2012 16:40:20 +0000

It was just over 2 years ago that I designed and stood up our first off-campus data center in Philadelphia, PA. Since that time we’ve completely vacated our original data center migrating all the servers, applications and services out to our new data center. Last month we relocated our offices leaving the old data center and office space behind forever. The new office space is very nice and has a lot of (very needed) conference rooms all of which have built-in audio/video capabilities with either an over-head projector or flat screen TV. I’m still hoping to have a LAN party someday on those 61″ monster displays perhaps with Call of Duty: Black Ops 2?

In June we started deploying our secondary data center with the intent of providing our own business continuity and disaster recovery services for our tier 1 applications including all our data storage needs. The design allows us the flexibility to utilize both DCs in an active/active configuration with the ability to move workloads (virtual machines) between DCs. While the design allows us that option we’re still testing how we’re going to handle all the different disaster scenarios – blade, enclosure, rack, SAN, cage, entire data center, etc. While our primary data center rings in at 800 sq ft our secondary data center is only 300 sq ft. This is possible because we’re utilizing a traditional disaster recovery model for our big box non-tier 1 applications that for one reason or another aren’t virtualized. This helps reduce the number of lazy assets hanging around and helps control some of the budget numbers. I totally expect the number of big box applications to continue to shrink over time as more and more application vendors embrace virtualization.

We’ve had pretty good success with the design of our first data center so we only made a few corrections. There’s a lot of logistics that need to be considered in any design especially around all the power and cooling requirements.

The Equipment

What equipment did we use? We already deployed Cisco at our primary data center so we decided to stay with Cisco at our secondary data center.

Cisco Nexus 7010
Cisco Nexus 5010
Cisco Nexus 2248
Cisco Nexus 1000V
Cisco Catalyst 3750X
Cisco Catalyst 2960G
Cisco ASA5520
Cisco ACE 4710
Cisco 3945 Router (Internet)
Cisco 2811 Router (internal T1 locations)

What racks did we use for the network equipment?

Liebert Knurr Racks
Liebert MPH/MPX PDUs

What equipment did we use for the servers/blades?

HP Rack 10000 G2
HP Rack PDU (AF503A)
HP IP KVM Console (AF601A)
HP BladeSystem C7000 Enclosure
HP Virtual Connect Flex-10 Interconnect
HP SAN 8Gb Interconnect
Cisco Catalyst 3120X
HP BL460c G7
HP BL620c G7
HP DL380 G8
HP DL360 G8

What are we using for storage?

IBM XIV System Storage Gen3 (SAN) (w/4 1Gbps iSCSI replication ports)
IBM SAN80B-4 SAN Switch
EMC DD860 (Disk-Disk backup via Symantec NetBackup)

Additional miscellaneous equipment;

MRV LX-4048T (terminal server)

We had some challenges with designing our secondary data center due to the density of our equipment. We had to stay under the maximum kw per sq foot load that the room (data center) was designed to handle. This is a simple calculation based on the kW utilization of the equipment to determine if there is adequate power and cooling available to meet that demand. We also had to maintain a N+1 design so we really can’t consuming more than 40% of our capacity leaving 10% for reserve. While some vendors charge a flat fee for the space (includes power) others charge per kWh so it’s very important to understand what type of demand you’re going to be placing on the data center.

My Design

We stood up a pair of Ciena 5200s from Zayo (formerly AboveNet) providing us a DWDM ring with 4 wavelengths between our primary data center and secondary data center . We’re using 2 wavelengths for the IP network between 2 pairs of Cisco Nexus 7010s and 2 wavelengths for the SAN fiber channel network between 2 pair of IBM SAN switches. We have the option of adding upwards of 4 additional wavelengths before we need to add any hardware so we have room for growth. The 4 wavelengths are diverse between an east and west path but they are not protected so it’s up to the higher layer protocols to provide the redundancy and failover.Not visible in the diagram above is a 10GE WAN ring that connects all our hospitals together. The primary and secondary data centers are also tied into that ring via multiple peering points for redundancy. You might be asking yourself why I’m using a Cisco 3750E as a termination switch in our primary data center. At the time we deployed our Cisco Nexus 7010s they didn’t support the 10GBase-ER SFP+ optic so I had to use the Cisco 3750E (with RPSU) as a glorified media transceiver/converter from 10GBase-ER to 10GBase-SR. The Cisco Nexus 7010 now has a 10GBase-ER SFP+ optic available so we didn’t need to use the Cisco 3750 in the secondary data center.

We are essentially stretching a Layer 2 vPC connection between the 2 data centers. It’s possible that some folks will get excited at the mention of Layer 2 between the data centers but it’s the best solution for us at this time and it certainly has pros and cons like everything in networking. We looked at potentially running OTV between the Cisco Nexus 7010s but ultimately decided to use a vPC configuration. We are only stretching the virtual machine VLANs that we need between the data centers.

My Thoughts

There’s a lot of work required to design any data center or even an ICR (Intermediate Communications Room), CCR (Central Communications Room), MDF (Main Distribution Frame) or IDF (Intermediate Distribution Frame). You’re immediately confronted with space, power and cooling challenges never mind coming up with the actual IP addressing scheme, VLAN assignments, routing vs bridging ,etc. You need to determine how much cabling you’ll need both CAT6 and fiber, perhaps you’ll look to use twinax of DAC (Direct Attach Copper) for your 10GE connections. Let’s not forget to include the ladder racks, basket trays, fiber conduits, PDUs, out-of-band networking, etc.

You also need to design the data center as if it was 300+ miles away… license those iLOs (HP Integrated Lights Out), purchase IP enabled KVMs, purchase console/terminal servers (Opengear or MRV) and wire everything up as if you will never have the opportunity to visit it again. We’ve had a few issues in the past few years that were quickly (less than 15 minutes) resolved thanks to having all our iLOs licensed, all our KVMs IP enabled, all our console/serial ports connected to a console/terminal server and the ability to dial-up into the console/terminal server should the problem get really bad.

Here’s a short story… We had a number of billing issues in the first few months of our contract with our current primary data center provider and the data from our Liebert PDUs, HP PDUs, and HP C7000 enclosures was invaluable in calling into question the numbers that were being reported to us. In all honesty when they told me we were consuming 53A on a 50A circuit I knew that something was grossly wrong with their math. In the end the provider admitted that there numbers were grossly wrong and the corrected numbers were in-line with the data we collected from our equipment.

It’s never a good idea to skimp on the documentation and I really advise taking lots of pictures, you’d be surprised how quickly you can forget what the back a specific rack looks like when you’re trying to walk Smart Hands through replacing a component at 2AM in the morning.

Cheers!