Michael McNamara

technology, networking, virtualization and IP telephony

Port Mirroring with ERS 8600

by

There was a recent comment asking for some additional information about the limitations of port mirroring on the Nortel Ethernet Routing Switch 8600. Once I started writing my reply I realized that it was probably big enough to stand as it’s own post.

I’ve taken the following definition straight from the “Network Design Guidelines (Part No. 313197-E Rev 00 June 2006)” ;

Port mirroring is a diagnostic tool that can be used for troubleshooting and performing network traffic analysis. When using port mirroring, you have to specify a destination port to see mirrored traffic and specify the source ports from which traffic is mirrored. Unlike other methods used to analyze packet traffic, packets flow normally through the destination port and packet traffic is uninterrupted.

For those Cisco folks in the audience port mirroring on a Cisco Systems switch is generally referred to as Switched Port Analyzer (SPAN) port.

There actually are quite a few different limitations and restrictions depending on the type of hardware you have in the ERS 8010 chassis and the version of software the switch is running. I’m going to limit myself to local port mirroring for this discussion but you can refer to my previous post on remote port mirroring.

  • Ingress mirroring mirrors packets that only have valid CRCs
  • Ingress mirroring is supported on all modules/cards
  • Egress mirroring is only supported on E or M modules/cards

icon-note Note: You can identify the type of modules/cards you have in your ERS 8600 with the “show sys info card” command from the CLI interface.

Legacy Modules (Non-E/E/M modules)

In software release 3.2.2 and later the following limitations apply;

  • The number of port mirroring entries that you can configure is between 1 and 383 and you can enable all entries simultaneously.
  • The number of mirroring ports plus the number of mirrored ports cannot exceed 384 (this is the maximum number of ports available in an ERS 8600 switch).
  • You can mirror ports supported by only the same OctaPID (group of eight 10/100 ports of a Gig port) to the same destination.
  • You cannot mirror one port.
  • You cannot mirror a port to multiple destinations.
  • You can configure a maximum of 64 destination ports at one time.

R Modules

On R modules you can configure one port mirroring entry for each lane on a module.

image

You can find a full list of OctaPID assignments at Nortel’s website.

Here are some examples. First let’s see what type of 8648TX card we have in slot 2;

ERS-8600:5# show sys info card

Card Info :

 ...

  Slot 2 :

        FrontType       : 48x100BaseTX-E
        FrontDescr      : TX48
        FrontAdminStatus: up
        FrontOperStatus : up
        FrontSerialNum  : SSCHE40FQM
        FrontHwVersion  : 05
        FrontPartNumber : 202572A31
        FrontDateCode   : 12212001
        FrontDeviations :

        BackType        : BFM6
        BackDescr       : BFM6
        BackSerialNum   : SSCHG70ETO
        BackHwVersion   : 05
        BackPartNumber  : 209536A11
        BackDateCode    : 12212001
        BackDeviations  :

It’s an E module so we can support both ingress and egress mirroring. Lets mirror port 2/48 to port 2/1 and we’ll place a packet sniffer (laptop with WireShark) on port 2/1.

ERS-8600:5# config diag mirror-by-port 1
ERS-8600:5/config/diag/mirror-by-port/1# create in-port 2/48 out-port 2/1 mode both enable true
ERS-8600:5/config/diag/mirror-by-port/1# info

Sub-Context:
Current Context:

                           create :
                           enable : true
                    mirrored-port : 2/48
                   mirroring-port : 2/1
                             mode : both
                           delete : N/A
            remote-mirror-vlan-id : 0

ERS-8600:5/config/diag/mirror-by-port/1# box
ERS-8600:5# show diag mirror

================================================================================
                         Diag Mirror-By-Port
================================================================================
ID   MIRRORED_PORT   MIRRORING_PORT  ENABLE     MODE       REMOTE-MIRROR-VLAN-ID
1    2/48            2/1             true       both       0

I generally find it’s a good idea to remove the destination port (mirroring-port) from any VLANs. This prevents broadcast traffic from that VLAN from contaminating the packet trace although you’ll still see STP BDUs since the port will still belong to a Spanning Tree group.

Cheers!

Ethernet Frames Maligned

by

I thought I would share this story with everyone. We had discovered an issue with Ethernet frames being maligned/corrupted between the Motorola Access Port 300 (AP300) and the Motorola Wireless (WS5100) LAN Switch.

We had a ticket open with Motorola trying to understand why a significant number of our AP300s were rebooting themselves at odd hours during the early morning. Motorola had requested that we provide network traces at the Access Point and Wireless Switch. Surprisingly Motorola came back and pointed out that the payload in some of the Ethernet frames was getting modified between the Wireless Switch and the Access Port.

The fundamental equipment involved in this problem were as follows; Nortel Ethernet Switch 460 (ES 460), Ethernet Switch 470 (ES 470), Ethernet Routing Switch 5520 (ERS 5520), Ethernet Routing Switch 8600 (ERS8600); Motorola Wireless LAN Switch 5100 (WS5100) and Access Ports 300(AP300).

The Motorola WS5100s and AP300s are physically connected over the same Layer 2 Ethernet network. The “Ethernet 1” port on the WS5100 is connected to a Virtual Local Area Network (VLAN) which provides a single broadcast domain for all AP 300s to connect to the WS5100. The “Ethernet 2” port on the WS5100 is used as a trunk interface to bridge between the WLANs (wireless) and VLANs (wired) segments. We essentially have core switches and edge switches (distribution is collapsed down into the core). The core switch can be a single ERS8600 or a pair of ERS8600s (Layer 3) connected via an IST (Inter-Switch Trunk). At the edge we generally deploy ES470(Layer 2) or ERS5520(Layer 2). We have deployed ES460s (PoE) into closets where ES470s are already present to specifically support PoE and the wireless network.

Here is a quick topology of the network with respect to the WS5100s and AP300s.
We recently started deploying the ERS5520s (in place of the ES470s) which directly support PoE allowing us to deploy one less piece of equipment at the edge and also provides one less bridge (hop) to switch through.We have been plagued by a problem that is affecting the Motorola AP300s causing them to randomly reset and re-adopt at different times of the day without warning or cause. In searching for the cause of this problem we’ve documented numerous Ethernet frames being maligned as they travel from the AP300 to the WS5100.

With respect to the examples I’m going to draw the following topology applies;

It should be noted that we do use the ES460s and ERS5520s to remark the 802.1p bits in the Ethernet frame so we can provide some measure of QoS with respect to the Nortel (Spectralink) Wireless LAN phones that we currently have deployed. In essence we mark all Ethernet packets on the “APVLAN” with a QoS level of 4 (“Gold”, BoSS-65530).

Network Trace Analysis

I will refer to the following two trace files;

“ers460side1.pcap” closet ES460 trace
“ers8600side1.pcap” core ERS8600 trace

I tried to merge up the two traces so each trace is synchronous with the other. We’ll focus on packet 3, you can see in the closet ES460 trace that bytes 15 and 16 are 0x20 and 0x12 respectively.

Looking at the other trace you can see that bytes 15 and 16 are different than in the first trace. You can see that the bits in 16 have been shifted to bytes 26.

You can again see the same problem in packet 4;


You can see it again in packets 6, 7, 10, 39, 43, 45, etc.

In the end the problem turned out to be a software/hardware issue with the Nortel Ethernet Routing Switch 8600. If DiffServ was enabled on the Ethernet port that was being mirrored, the mirrored data was somehow getting corrupted in the process of copying the packets. Once we disabled DiffServ on the Ethernet port the problem disappeared. We opened a case with Nortel but were told that it would be handled as an enhancement request, not a correction request (go figure!).

I personally no longer trust either the port mirror or packet capture facilities of the Nortel ERS 8600 and rely on physical taps so there can be no doubt or questions about the validity of the capture data.

We still have issues with our Motorola AP300s rebooting from time to time but they have been much better since Motorola released v2.1.3 software for the WS5000/WS5100s. We are currently working with Motorola to resolve issues in their v3.x software line that is causing our Nortel 2211 (Spectralink) wireless phones to occasionally reboot while idle and roaming.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL