Michael McNamara

technology, networking, virtualization and IP telephony

Cisco Layer 2 Switching with Multicast and IGMP Snooping

by

I recently happened upon a familiar problem with IGMP Snooping on a Layer 2 topology comprised of Cisco Catalyst 6504 and 4948 switches. Another team was having issues getting Multicast traffic to pass between their Xen hosts which were all on the same VLAN, but where physically wired to the two different switches mentioned above. There was a trunk interface between the two switches, passing all the VLANs so there was nothing wrong with the basic Layer 2 forwarding. In general Multicast frames will be flooded across all ports in the VLAN, unless IGMP snooping is enabled which it is by default in Cisco switches. I remember quite a few challenges with IGMP snooping back in the Nortel and Avaya days. Avaya eventually changed their default configuration such that IGMP snooping is now disabled by default.

In this specific case all the routing was being performed by a number of high-end Cisco ASA firewalls which didn’t have PIM routing configured or enabled so I took the easy approach of just disabling IGMP snooping across the Cisco Catalyst 6504 and 4948 switches and the problem was solved. The cleaner solution would have been to setup an Mutlicast Router (mrouter) on the VLAN to properly handle all the IGMP requests and reports.

As pointed out by a colleague you can use a great little Python script written by RedHat for testing Multicast on your Linux servers.

Cheers!

Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 3 of 30. All the posts can be viewed from the 30in30 tag.

Reference;
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/68131-cat-multicast-prob.html
http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/configuration/guide/config/route_multicast.html

Avaya Contact Center Agent Desktop Display Quietly Crashing

by

I thought I would share this story… it’s another story of “it’s the network’s fault” when in reality it really has nothing to-do with the network but it falls to the network engineers and consultants to prove the point beyond a reasonable doubt.

I can’t tell you how it irks me to hear people say “it’s the networks fault” when they have absolutely no clue as to how anything works and have no data to support their wild claims. I would think a lot more of them if they just said, “I’m sorry, I haven’t got a frigging clue what’s happening here but can you help me?” And of course the problem always needs to be resolved yesterday as if the building itself was on fire.

We have multiple Avaya Aura Contact Center (formerly Nortel Symposium) installations. At one of these locations we began receiving trouble tickets that the Agent Desktop Display (ADD) which is a small software application that listens to a Multicast stream and displays a ticker tape banner showing the contact center queue details was quietly closing after only a few minutes of running on the local desktop/laptop. The local telecom technician verified that the problem only occurred on a specific floor, the users on the other floors had no issues or problems. A quick check of the core Avaya Ethernet Routing Switch 8600 and edge Avaya Ethernet Routing Switch 5520s indicated that IGMP and PIM were configured and working properly.

Note:A few years back now I detailed how to configure IGMP, DVMRP and PIM for Multicast routing.

I asked the local telecom technician to perform a packet trace so I could see what was happening on the wire. The packet trace indicated that the desktop/laptop was issuing an IGMP leave request and was closing the HTTP/TCP socket it had open to the web server so that was proof enough for me that the application was silently crashing and the operating system was cleaning up all the open ports and IGMP sessions.

6376 2013-05-16 07:47:46.052281 10.1.46.144 10.1.38.55 TCP     54   3317 > 80 [RST, ACK] Seq=4467 Ack=10430 Win=0 Len=0
6377 2013-05-16 07:47:46.052595 10.1.46.144 224.0.0.2  IGMPv2  46   Leave Group 230.0.0.2

The actual Multicast stream from the application/web server was fine;

6353	2013-05-16 07:47:43.995183	10.1.38.55	230.0.0.2	UDP	511	Source port: 1031  Destination port: 7040
6354	2013-05-16 07:47:43.995502	10.1.38.55	230.0.0.2	UDP	502	Source port: 1025  Destination port: 7050
6355	2013-05-16 07:47:43.995885	10.1.38.55	230.0.0.2	UDP	813	Source port: 1026  Destination port: 7030
6356	2013-05-16 07:47:43.996301	10.1.38.55	230.0.0.2	UDP	860	Source port: 1032  Destination port: 7020
6357	2013-05-16 07:47:43.996505	10.1.38.55	230.0.0.2	UDP	343	Source port: 1033  Destination port: 7060
6358	2013-05-16 07:47:43.996726	10.1.38.55	230.0.0.2	UDP	331	Source port: 1027  Destination port: 7070
6359	2013-05-16 07:47:43.996886	10.1.38.55	230.0.0.2	UDP	153	Source port: 1028  Destination port: 7110
6360	2013-05-16 07:47:43.997048	10.1.38.55	230.0.0.2	UDP	153	Source port: 1034  Destination port: 7100
6361	2013-05-16 07:47:43.997199	10.1.38.55	230.0.0.2	UDP	135	Source port: 1030  Destination port: 7090
6362	2013-05-16 07:47:43.997371	10.1.38.55	230.0.0.2	UDP	135	Source port: 1036  Destination port: 7080
6363	2013-05-16 07:47:43.997525	10.1.38.55	230.0.0.2	UDP	127	Source port: 1035  Destination port: 7120
6364	2013-05-16 07:47:43.997647	10.1.38.55	230.0.0.2	UDP	127	Source port: 1029  Destination port: 7130

The packet trace did show some odd UDP broadcast traffic from one specific desktop that happen to be running GE’s Centricity Perinatal (CPN). This is a software application used to monitor Labor & Delivery, the Nursery and the NICU. We use it to actually monitor, chart and graph the strips put out by the fetal monitors. There’s a software component of the GE CPN solution called B-Relay which is the piece of software that floods the VLAN with all those UDP broadcasts. Unfortunately this UDP flooding is by design and is required for the application to function properly.

6205	2013-05-16 07:47:26.710685	10.1.47.210	10.1.47.255	UDP	251	Source port: 1759  Destination port: 7005
6206	2013-05-16 07:47:26.853810	10.1.47.210	10.1.47.255	UDP	822	Source port: 1760  Destination port: 7043
6211	2013-05-16 07:47:28.215486	10.1.47.210	10.1.47.255	UDP	60	Source port: 1783  Destination port: 7013

Looking at the packet traces I quickly noticed that while there are multiple destination ports they are overlapping between 7001 and 7999. I would theorize that the GE CPN software was eventually hitting a UDP port that the ADD software was listening on and since it was a broadcast packet it tried to process the data and was quietly choking and crashing. I shutdown the Ethernet port connecting the GE CPN desktop and had the local telecom technician run his test again. He called back about 30 minutes later to let me know that everything was working fine and that whatever I had done had fixed the problem. Well it wasn’t really fixed because now I had to figure out how to get both applications to co-exist.

The solution was to isolate the GE CPN desktops to their own VLAN so that the UDP broadcasts wouldn’t hit the closet VLAN where the Contact Center users resided. Another possible solution might have been to try and change the UDP ports that either GE CPN or Avaya ADD software was using but that change would have probably taken weeks if not months. I was able to spin up a new VLAN in about 30 minutes and get everyone back up and running again.

Have you got a story to share? I’d love to hear it!

Cheers!

Multicast Routing Protocol (Part 2)

by

In part 1 of this post I looked at how to configure DVMRP to facilitate inter-VLAN Multicast commuications on a single switch. In this post I’ll look at how to configure PIM to facilitate inter-VLAN Multicast communications across multiple switches and routers (Layer 3 switches).

I took a few minutes and threw together a quick diagram to help layout the topology (a picture is truly worth a thousand words). There are two core ERS 8600 switches (a switch cluster as Nortel likes to call it these days). There are three VLANs bridged across all four switches in the diagram, VLAN 55, 56 and 200. There is a fourth VLAN, 57, that is routed from ERS 8600 C. The ERS 5520 in the diagram will only be used as a Layer 2 even though it could potentially be used as a Layer 3 device (router).


I’m going to review two possible configurations. The first scenario will be for a client device (VLC Client A) in a VLAN routed by the core ERS 8600s. The second scenario will be for a client device (VLC Client B) in a VLAN routed by a closet ERS 8600.

Lets get on with configuring some ERS 8600 switches. First lets enable PIM globally;

ERS8600-A# config ip pim enable
ERS8600-A# config ip pim fast-joinprune enable

Then we’ll enable PIM on the specific VLANs;

ERS8600-A# config vlan 55 ip pim enable
ERS8600-A# config vlan 56 ip pim enable
ERS8600-A# config vlan 200 ip pim enable

We need to create a CLIP interface to use for PIM routing, we don’t want to tie the PIM routing to a physical interface in case that interface goes down for whatever reason. We’re already using CLIP 1 for our OSPF router ID of 10.1.0.5/32.

ERS8600-A# config ip circuitless-ip-int 2 create 10.1.0.15/255.255.255.255
ERS8600-A# config ip circuitless-ip-int 2 ospf enable
ERS8600-A# config ip circuitless-ip-int 2 pim enable

We need to add a candidate Rendezvous Point Router (RP) pointing it to our CLIP address.

ERS8600-A# ip pim candrp add grp 239.255.1.1 mask 255.255.255.255 rp 10.1.0.15

We need to set the priority of the Bootstrap Router (BSR) for dynamic PIM routing.

ERS8600-A# ip pim interface 10.1.0.15 cbsrpreference 100

Then on the second core ERS 8600 switch;

ERS8600-B# config ip pim enable
ERS8600-B# config ip pim fast-joinprune enable
ERS8600-B# config vlan 55 ip pim enable
ERS8600-B# config vlan 56 ip pim enable
ERS8600-B# config vlan 200 ip pim enable
ERS8600-B# config ip circuitless-ip-int 2 create 10.1.0.16/255.255.255.255
ERS8600-B# config ip circuitless-ip-int 2 ospf enable
ERS8600-B# config ip circuitless-ip-int 2 pim enable
ERS8600-B# config ip pim candrp add grp 239.255.1.1 mask 255.255.255.255 rp 10.1.0.16
ERS8600-B# config ip pim interface 10.1.0.16 cbsrpreference 50

That’s really all there is to configure with the two core ERS 8600 switches.

ERS5520 Switch (Edge)
In the case of the ERS 5520 switch there really isn’t anything you need to configure per say. You could enable IGMP (generally disabled by default) to filter the multicast traffic from ports that are not subscribing to any multicast groups. Since the ERS8600s are performing the routing the ERS5520 acts just like a Layer 2 switch.

VLC Client A (10.1.56.50) should now be able to connect to the multicast group 239.255.1.1 from the ERS 5520 which will be sourced from the VLC Server (10.1.55.50).

ERS8600 C Switch (Edge)
In the case of the ERS 8600 switch (edge) you need to configure and enable PIM. We’ll using VLAN 200 to interface with the upstream ERS8600 switches

ERS8600-C:5# config ip pim enable
ERS8600-C:5# config vlan 57 ip pim enable
ERS8600-C:5# config vlan 57 ip pim interface-type passive
ERS8600-C:5# config vlan 200 ip pim enable

Since there won’t be any other Layer 3 PIM switches on VLAN 57 we set the PIM interface to passive (much like the OSPF equivalent of passive).

VLC Client B (10.1.57.50) should now be able to connect to the multicast group 239.255.1.1 from the ERS 8600 C which will be sourced from the VLC Server (10.1.55.50).

We can dump the multicast (PIM) routing table with the following command from the edge ERS8600 switch;

ERS8600-C:5# show ip pim mroute

================================================================================
Pim Multicast Route
================================================================================
Src: 0.0.0.0 Grp: 230.0.0.2 RP: 10.1.0.5 Upstream: 10.1.200.5
Flags: WC RP CACHE
Incoming Port: Vlan200-1/1,
Outgoing Ports: Vlan127-2/42,
Joined Ports:
Pruned Ports:
Leaf Ports: Vlan127-2/42,
Asserted Ports:
Prune Pending Ports:
Assert Winner Ifs:
Assert Loser Ifs:
TIMERS:
Entry JP RS Assert
151 1 0 0
VLAN-Id: 200
Join-P: 0
Assert: 0
——————————————————————————–
Src: 10.1.233.30 Grp: 230.0.0.2 RP: 10.1.0.5 Upstream: 10.1.200.5
Flags:
SPT CACHE SG
Incoming Port: Vlan200-1/1,
Outgoing Ports: Vlan127-2/42,
Joined Ports:
Pruned Ports:
Leaf Ports: Vlan127-2/42,
Asserted Ports:
Prune Pending Ports:
Assert Winner Ifs:
Assert Loser Ifs:
TIMERS:
Entry JP RS Assert
64 4 0 0
VLAN-Id: 200
Join-P: 0
Assert: 0
——————————————————————————–

Total Num of Entries Displayed 2
Flags Legend:
SPT = Shortest path tree, WC=(*,Grp) entry, RP=Rendezvous Point tree, CACHE=Kernel Cache, ASSERTED=Asserted, SG=(Src,Grp) entry, PMBR=(*,*,RP) entry, FWD_TO_RP=Forwarding to RP, FWD_TO_DR=Forwarding to DR, SG_NODATA=SG Due to Join, CP_TO_CPU=Copy to CPU, STATIC_MROUTE=Static Mroute, MRTF_SMLT_PEER_SG=Peer SG On Non-DR For SMLT
——————————————————————————–

Troubleshooting

Here are some basic commands that should help you troubleshoot any PIM issues;

ERS8600-A:5# show ip pim neighbor

================================================================================
Pim Neighbor
================================================================================
INTERFACE ADDRESS UPTIME EXPIRE
——————————————————————————–
Vlan55 10.1.55.6 31 day(s), 00:09:53 0 day(s), 00:01:40
Vlan56 10.1.56.6 31 day(s), 00:09:53 0 day(s), 00:01:40
Vlan200 10.1.200.6 31 day(s), 00:09:53 0 day(s), 00:01:34

Total PIM Neighbors = 3

We can see that all three VLAN interfaces have PIM neighbors with the ERS 8600 B switch. Lets just check the RPs and make sure we have the correct multicast groups (addresses).

ERS8600-A:5# show ip pim rp-set

================================================================================
Pim RPSet
================================================================================
GRPADDRESS GRPMASK ADDRESS HOLDTIME EXPTIME
——————————————————————————–
230.0.0.1 255.255.255.255 10.1.0.15 150 137
230.0.0.2 255.255.255.255 10.1.0.15 150 137
239.255.1.1 255.255.255.255 10.1.0.15 150 137

The multicast addresses of 230.0.0.1 and 230.0.0.2 listed above are used for Nortel’s Contact Center (formerly Symposium Call Center software). Here’s how we can list the candidate RPs;

ERS8600-A:5# show ip pim candidate-rp

================================================================================
Pim Candidate RP Table
================================================================================
GRPADDR GRPMASK RPADDR
——————————————————————————–
230.0.0.1 255.255.255.255 10.1.0.15
230.0.0.2 255.255.255.255 10.1.0.15
239.255.1.1 255.255.255.255 10.1.0.15

If we’re dynamically choosing a RP we need to make sure that there is a BSR active;

ERS8600-A:5# show ip pim bsr

================================================================================
Current BootStrap Router Info
================================================================================

Current BSR address: 10.1.0.15
Current BSR priority: 100
Current BSR HashMask: 255.255.255.252
Current BSR Fragment Tag: 44590
Pim Bootstrap Timer : 31

I may need to update this article to make it cleaner and clearer.

Cheers!

Multicast Routing Protocol (Part 1)

by

I was originally just going to write about DVMRP, but I’ve also decided to post some basic examples for setting up PIM-SM. I’ll break this post into two parts; first part will look at utilizing DVMRP to setup a simple Multicast domain on a single switch while the second part will look at utilizing PIM-SM across multiple switches.

We have a few Nortel Contact Center (formerly Symposium) installations deployed throughout the organization. The Nortel Agent Desktop Display (ADD) utilizes multicast to distribute the information between the server and the individual clients. Unless the clients are in the same VLAN as the server (Application/Web server and Database server) you’re going to need a Multicast Routing Protocol to facilitate the multicast communications between VLANs. I should point out that at this point I’m only talking about making multicast traffic available between VLANs on a single Nortel Ethernet Routing Switch 8600.

Note: Nortel Contact Center 6.0 appears to use the following two Multicast addresses by default; 230.0.0.1, 230.0.0.2

Unfortunately I didn’t have a spare Contact Center server to test with so I needed to figure out how I could test multicast traffic ahead of time and then just schedule any changes that needed to be made to facilitate inter-VLAN multicast communications. I recalled that VideoLAN – VLC media player could stream audio/video via multicast.

In order to test I setup two laptops running Windows XP Service Pack 2, laptop A (10.1.55.50/24) on VLAN 55 (10.1.55.0/24) and laptop B (10.1.56.50/24) on VLAN 56 (10.1.56.0/24).

Laptop A will be the broadcast server and stream the video while laptop B will be the client.

Let’s setup the ERS 8600 switch;

ERS-8610:6# config vlan 55 create byport 1
ERS-8610:6# config vlan 55 ip address 10.1.55.5/24
ERS-8610:6# config vlan 55 ip ospf enable
ERS-8610:6# config vlan 55 ip vrrp 1 10.1.55.1
ERS-8610:6# config vlan 55 ip dvmrp enable
ERS-8610:6# config vlan 56 create byport 1
ERS-8610:6# config vlan 56 ip address 10.1.56.5/24
ERS-8610:6# config vlan 56 ip ospf enable
ERS-8610:6# config vlan 56 ip vrrp 1 10.1.56.1
ERS-8610:6# config vlan 56 ip dvmrp enable

And then some global settings;

ERS-8610:6# config ip dvmrp enable
ERS-8610:6# config ip ospf enable

Now we need to look at how to make VLC do what we need;

Once you install VLC and start the program you will be greeted by this lightweight frontend.

Click File -> Open File to bring up the Open dialog box.

Click on the Browse button to bring up a standard Windows file selection box. Select the file you want to play. Then click Open.

Your selection should appear in the text box next to the Browse button. Click the check box for Stream Output and then click the button Settings.

If you wish to view the video on the source laptop then check the box next to Play Locally under Output Methods. When streaming to another system you don’t have to play the file on the server, but you can use this option to visually confirm that our video is playing properly before trying to access the stream from another computer.

Check the box marked UDP and type in the Muticast address you want to stream the file to. You should use a local-scope multicast address between 239.0.0.0 – 239.255.255.255. You should also make sure that the Time-To-Live (TTL) is set to 2. Then click OK. The file is ready to play so click OK in the Open dialog box too.

The video or audio file should begin playing on the computer. The last thing to do before switching to the second laptop is to turn on VLC’s web interface by clicking Settings -> Add Interface -> Web Interface. This will help provide remote control over VLC if we should need it from the second laptop.

Open VLC on the second laptop.

Click on File -> Open Network Stream. Select UDP/RTP Multicast and use the same Multicast address you use on the server. Click the OK button and VLC will start playing your stream.

Now that the stream is successfully playing on your computer you can open up a web browser to control VLC remotely. Type http://10.1.55.10:8080/ into the address bar. The web browser will present you with all of the controls you need to manage playlists and playback remotely.

If you’ve setup the ERS8600 properly your video should start playing on the client laptop.

If you want to make sure that VLC is configured and working properly move both laptops to the same VLAN. If the video stream works then you know that VLC is working properly and you need to focus the network configuration.

Note: Windows XP defaults to IGMP v3 which is fine for this test.

You can use the following commands to troubleshoot the network pieces. In the examples below I had the laptops connected to an ERS 5520 switch which was uplink on port 1/1. That is why the port is reported as 1/1 throughout the different commands.

DVMRP

ERS-8610:6# show ip dvmrp info
==================================================================                        Dvmrp General Group
==================================================================

AdminStat               : enabled
Genid                   : 0x47c42ef1
Version                 : 3
NumRoutes               : 2
NumReachableRoutes      : 2

UpdateInterval          : 60
TriggeredUpdateInterval : 5
LeafTimeOut             : 125
NbrTimeOut              : 35
NbrProbeInterval        : 10
FwdCacheTimeout         : 300
RouteExpireTimeout      : 140
RouteDiscardTimeout     : 260
RouteSwitchTimeout      : 140
ShowNextHopTable        : disable
generate-trap            : disable
generate-log             : disable
PruneResend             : disable

ERS-8610:6# show ip dvmrp interface

================================================================================                        Dvmrp Interface
================================================================================                                         DEFAULT DEFAULT DEFAULT ADVERTISEIF        ADDR            METRIC OPERSTAT LISTEN  SUPPLY  METRIC  SELF
-------------------------------------------------------------------------------
Vlan55    10.1.55.1       1      up       enable  disable 1       enable
Vlan56    10.1.56.1       1      up       enable  disable 1       enable

2 out of 2 entries displayed

--------------------------------------------------------------------------------
IF        ADDR            IN-POLICY       OUT-POLICY      INTF TYPE
--------------------------------------------------------------------------------
Vlan55    10.1.55.1                                      ActiveVlan56    10.1.56.1                                      Active

2 out of 2 entries displayed

ERS-8610:6# show ip dvmrp route

================================================================================
                       Dvmrp Route
================================================================================
SOURCE          MASK            UPSTREAM_NBR    INTERFACE  METRIC EXPIRE
--------------------------------------------------------------------------------
10.107.55.0     255.255.255.0   0.0.0.0         Vlan55     1      155
10.107.56.0     255.255.255.0   0.0.0.0         Vlan56     1      155

2 out of 2 entries displayed

IGMP 

ERS-8610:6# show ip igmp cache
================================================================================
                        Igmp Cache
================================================================================
GRPADDR         INTERFACE  LASTREPORTER    EXPIRATION V1HOSTTIMER  TYPE       STATICPORTS
--------------------------------------------------------------------------------
239.255.1.1     Vlan56     10.1.56.50    213        0            DYNAMIC NULL
239.255.255.250 Vlan55     10.1.55.50    214        0            DYNAMIC NULL
239.255.255.250 Vlan56     10.1.56.50    219        0            DYNAMIC NULL

3 out of 3 entries displayed

ERS-8610:6# show ip igmp group

================================================================================
                        Igmp Group
================================================================================
GRPADDR         INPORT          MEMBER          EXPIRATION TYPE
-------------------------------------------------------------------------------
239.255.1.1     V56-1/1         10.1.56.50      209        Dynamic
239.255.255.250 V55-1/1         10.1.55.50      210        Dynamic
239.255.255.250 V56-1/1         10.1.56.50      215        Dynamic

Total number of groups 3Total number of unique groups 2

ERS-8610:6# show ip igmp sender

================================================================================
                        Igmp Sender
===============================================================================
GRPADDR         IFINDEX    MEMBER          PORT       STATE
--------------------------------------------------------------------------------
239.255.1.1     Vlan 55    10.1.55.50      1/1        NOTFILTERED

1 out of 1 entries displayed

Hopefully I haven’t gone over the top on this one.

Please post any comments, corrections or suggestions.

Cheers!

Recent Posts

Categories

SUBSCRIBE BY EMAIL