You can certainly “merge” the VLANs between the two solutions… assuming this is a temporary solution until you remove one or the other you can just setup a simple SMLT trunk between the two. I would just set it up as a simple SMLT configuration on each side, remember the two clusters have no idea what they are connecting to so from that perspective it’s just a simple LACP over SMLT configuration.

Cheers!

I am a bit new to Avaya Data platform, so need your guidance for a proposed network changes in our setup. I am not sure if this is the right forum for that.

We have two different networks with different set of Cores ( VSP 8400 with RSMLT in one setup & ERS 8800 with RSMLT in second setup ). Now we have a requirement of merging these two networks. I know that two switch clusters of same type can be merged, but not sure if switch clusters of two different models can be merged or not. If you could confirm that , it will help.

Also, this is our requirement after the merging. please advise if these can be achieved by merging two clusters.

1. VLANs of Cluster 1 should be able to access some VLANs of cluster 2 & vice versa. Do we need to create same set of VLANs on both clusters? or routing them using RSMLT port configuration will work.
2. Static routes configured on both the clusters should still be able to work as before. Is this workable?

Please advise.

Regards,

Rajesh

Hi Ramkumar,

You are correct… you want to set the holdup timer to 9999.

Cheers!

Your document was very crisp and useful but in case of edge deployment can we leave the holdup timer by default (180) , because when Core A is completely down , core B will forward the traffic destined for Core A only for 180 seconds right ? I read some documents saying it should be 9999 (infinite ) isn’t it ?

Regards,

Ram

You should either run VRRP or RSMLT but not both at the same time.

It’s not a simple check box… you most likely need to re-address your Layer 3 (IP) interface.

See comment #8 above for a script that I used to make the change.

Cheers!

i have VRRP configured in my netwotk with back-up master enabled for 8600. do i need to disable it prior configuring RSMLT.

Sayantani

I’ve done a simple Visio diagram, that makes the descriptive stuff above, more meaningful.

I’m happy to email it to you if you’d like to put it up.

Cheers…

Dennett

Thanks for the update Dennett.

It’s been a while since I’ve worked with a PIM-SM Multicast configuration so some of the details are a little fuzzy but I can follow your logic.

Would love to hear the ultimate outcome. I might try to set this up in my lab just to confirm but my time is very tight these days.

Any thoughts about making this into a guest post? How to setup PIM-SM on and ERS 8600 switch cluster for HA?

Thanks for sharing!

I tested things in the lab today and thought I’d let you know and your readers how it went.

A bit of background about the reason Multicast fails over RSMLT and over VRRP when the Master 8600 dies.

The lab test 5530 distribution switches use a default unicast route pointing to 8600A(10.10.10.1) for Layer 3.

When 8600A(10.10.10.1) fails, 8600B(10.10.10.2) assumes responsibility for 8600A’s(IP/MAC) – RSMLT at work.

This is fine for Layer 3 but Multicast routing is a bit more complex.

The multicast routing (Mroute) table is (in my case with STATIC configuration) built from the information in the unicast routing table. Because I only have a single default route, there will only be one entry in my Mroute table.

The important bit of information in this table, is the RPF (Reverse Path Forwarding) neighbor. In my case (10.10.10.1) again.

PIM-SM also keeps a PIM neighbors table on each device, which is populated in the 5530’s case, by PIM Hello messages sent from the 8600’s every 15 secs.

My test 5530 stack, has two PIM neighbor entries, one for 10.10.10.1 and one for 10.10.10.2 (These entries expire after 90 seconds if no Hello message has arrived)

When you have a Multicast Source or Client on your switch, a Join/Prune message is generated by the switch and forwarded to each RPF neighbor in the Mroute table. BUT – “O N L Y” if there is a corresponding entry for that IP in the PIM neighbor table. See the PIM-SM RFC…

When 8600A(10.10.10.1) goes down, it’s record in the PIM neighbor table expires after 90 seconds and disappears. (No more PIM Hello’s…)

Now when the 5530 builds the Join/Prune message, it checks the Mroute table (we have only one entry for 10.10.10.1), which it uses, then looks for 10.10.10.1 in the PIM neighbor table. It is no longer there, so the message is never sent. See RFC again…

There will never be a Join/Prune built for 8600B(10.10.10.2) because there is no Mroute entry for 10.10.10.2 in our table.

Multicast therefor fails.

As an aside, the reason multicast fails on VRRP is similar. The PIM Hellos from the 8600’s come from the physical addresses, not from the VRRP VRIP.

So there will never be an entry in the PIM neighbor table for a virtual IP address.

OK – How Do We Fix This (for STATIC environments)?

We could configure a STATIC Mroute entry for 10.10.10.2 with the CLI command config ip static-mroute create rpf
[preference ]

Now an RPF Neighbor entry for 10.10.10.2 is in our Mroute table and our PIM Neighbor table and a Join/Prune can be built and sent to 10.10.10.2

Multicast now continues if 8600A dies.

UNFORTUNATELY, STATIC Mroute entries are NOT supported by the 5500 series switches, so –

We configure a virtual PIM Neighbor entry for 10.10.10.1 with the CLI command ip pim virtual-neighbor 10.10.10.3 10.10.10.1

This is done in the 5530 (10.10.10.3).

Now there will always be an entry for 10.10.10.1 in the PIM Neighbor Table and the Join/Prune message can be successfully sent. (10.10.10.1 is in our Mroute table and in our VirtPIM Neighbor table).

Multicast now continues if 8600A dies.

For VRRP if your VRIP was 10.10.10.4 you would ip pim virtual-neighbor 10.10.10.3 10.10.10.4

I will be testing ECMP as an alternative next week and update if you are interested.

I hope this helps someone. And hope I’ve made no typos. It was an interesting problem and fun to debug…

I could do I diagram and email it to you to include, Michael. Just let me know.

Cheers…

Dennett

I think I may have found the cause of the problem for the multicast failure over RSMLT and perhaps the reason why multicast fails over VRRP.

I will do some more testing in the lab tomorrow and will post my results. They may help someone else.

Thanks for your input and interest …

Cheers…

Dennett

No, there are no 5530’s with IST.

They are all stacked DMLT, back to RSMLT on the core 8600’s.

From the 5530’s downstream – MLT to the 4548’s.

Cheers…

That’s a nice sized network, I won’t dare to ask how many actual ports if you have 30,000 users.

Just a quick word of warning, if you are running the ERS 5530s in an IST/SMLT cluster configuration then PIM-SM is not supported on those switches, only IGMP. You’ll need another switch/router to run the PIM-SM functionality.

There were several discussions over on the forums that might not be of too much value but you might be interested in reviewing anyways.

Let us know how you make out if you have the time.

Good Luck!

Thanks again, I’ll read all of that.

just for clarity the 5530’s are Layer 3. (Distribution switches)

No have not tried dynamic BSR, trying to keep everything static if possible but this may not be practical.

There are also be layer 2 (edge switches) downstream from these, which I will also add to the lab testing.

Production is more complex (wish I could put up a diagram) – 8 x 8600’s full mesh, layer 3 distribution switches (5530’s), layer 2 edge switches (4548’s). (Too many to count) – around 30,000 users.

The 5530’s will be the RP’s for each building (around 55 buildings).

Just need to be sure this is going to work before I commit to production.

Dennett

Here are some additional documents that might help shed some light on the issue;

NN46205-200_02.02_Planning-Engineering_Network-Design.pdf
pp8600_tcg_pim_sm_public.pdf

You’ll find the PIM-SM document is a little old but should still serve as a good resource.

Cheers!

Hi Dennett,

Thanks for the kind comments.

I believe I now understand the problem you are experiencing.

Are the edge switches Layer 2 or Layer 3?

Have you tried your configuration with a dynamic BSR?

Also you could probably tie your RPs to a CLIP interface and not the physical IP interface for any specific VLAN. I did find an interesting note in a post I made a while back regarding the 4.1.8.2 software release; Not all IP Multicast failover or fail-back/recovery situations can provide such times today, as many situations depend on the IPMC protocol recovery. For best IPMC recovery in SMLT/RSMLT designs, the use of static RPs for PIM-SM is recommended, with the same CLIP IP address assigned to both Core IST Peers within the Cluster, and to all switches with a full-mesh or square configuration.

If you look at my post titled, Multicast Routing Protocol (Part 2), you’ll see the example I provided used the CLIP for the RPs.

I believe your configuration is supported. If you look at this document, 2008_07_21_Switch_Clustering_Supported_Topologies_and_Interoperability_with_ERS_NN48500-555_Version1200.pdf page 8, specifically example #6 assuming your edge switches are Layer 2.

I’ll see if I can ask around but I know there were some very LARGE Multicast users over in the EU and that Nortel/Avaya had devoted significant resources to resolves all the known issues in both 4.1.8.3 and ultimately in 5.1.3.0.

Good Luck!

Thanks for the reply and great site by the way…

I probably should have provided more info in my first post but wasn’t sure this thread would still be active after a year.

There is no problem with the config of either of the clustered 8600’s nor with the 5530’s. PIM remains enabled for all VLAN’s and the RP’s are fine. The entire configuration on all devices by the way is STATIC. Routing, RP’s etc, etc.

The simple fact that getting Mulicast working again on the backup 8600 (.2), is a simple as entering a static L3 route into the 5530’s pointing to (.2) as the gateway to the RP’s, tells me that all of the PIM-SM info is correct and working.

It is just that Multicast does not like it, when (.2 backup) assumes the (.1 IP and MAC) when the master 8600 is taken down. I am doing all of this for testing prior to production implementation. But at the moment it doesn’t look good for Multicast.

Multicast over VRRP has similar (but more varied) failures. RSMLT is supposed to have fixed this.

I’m using 5.1.2 on the 8600’s and 6.2.200 on the 5530’s.

The last one is not GA.

Thanks…

Hi Dennett,

You’re probably forgetting to re-enable PIM on that specific VLAN. When you delete the IP interface the switch will remove all protocols that are dependent on that IP interface. That’s why you need to re-enable DHCP and then re-create the DHCP forwarding agent. Likewise if you look at the PIM (PIM-SM) tab for that VLAN you’ll probably find that it’s unchecked (disabled). You’ll need to enable PIM and you might need to recreate some of your PIM configuration such as the RPs. You should probably look at your old configuration (prior to any changes) and then just cut-n-paste the related PIM commands back into the switch.

Good Luck!

I notice this article is about year old and I had already configured RSMLT exactly as described above and it has been working perfectly for Layer3 for some time. I have just this week enabled Multicast PIM-SM on this pair of 8600’s feeding multiple 5530 stacks running DMLT connected to the 8600’s.

The problem I have is with Multicast only – but exactly as described in your post. Layer 3 returns almost immediately but Multicast never recovers on it’s own.

I need to bring back up the (.1) 8600 which is the default gateway, or for “test purposes only – to see the behaviour”, change the (.2) 8600 address to (.1) or add a static route to the 5530’s (the RP’s) via the (.2) 8600.

All of which re-establish proper multicast operation.

The last two are not able to be done in practice – obviously.

At no time is layer 3 affected, it handles the loss flawlessly. It’s just multicast that gets screwed.

Any insights?

Cheers…

You won’t have any issues with your edge switches but you will have some brief routing/reachability issues. You should expect some brief routing/reachability issues on the affected VLANs while you’re making the changes. Once you’ve restored the IP interface and OSPF has converged you should be good to go.

I’d also recommend that you reboot (yes I said reboot) your ERS 8600 after you’ve made all the changes and everything looks good. I’ve found that a reboot/restart/cpuswitchover really helps provide some stability to the environment after a large number of changes such as those being discussed here.

You also need to make sure you enable RSMLT edge support (documented above) and let’s not forget to enable RSMLT globally and not just on the individual VLANs.

With regard to IGMP and DVMRP you’ll need to check what happens once you delete the IP interface. I haven’t tried it myself but I suspect that without an IP interface IGMP and DVMRP might become disabled as does the OSPF interface and the DHCP relay agent.

Good Luck!

First of all thank you for the script. It will really help. Secondly let me recap and correct me if I am wrong:
1. Disable OSPF
2. Delete old and create new interface
3. Enable OSPF
3. Enable RSMLT at the Core
4. Enable DHCP forwarding
5. I am running IGMP with DVMRP – do I have to disable multicast and reenable after?
Any issues with BPS stacks ?

Adrian

Hi Adrian,

It wasn’t very hard at all… I actually took some time to write an AWK script to help remove any “fat-finger” issues.

Let me dig up that script right now… even if you don’t care to use the script it will give you a good idea of the steps necessary.

#!/bin/awk -f
#
###################################################################
###################################################################
#
# We need to gather the input data from the CLI interface of the
# switch. You’ll need the VID, OLD IP, NETMASK and NEW IP.
#
# Here’s the data file format that the AWK script expects;
#
# VID OLD IP NETMASK NEWIP
# 1 10.124.20.201 255.255.240.0 10.124.16.1
# 3 10.124.80.1 255.255.240.0 10.124.80.1
# 4 10.124.160.1 255.255.240.0 10.124.160.1
# 5 10.124.176.1 255.255.240.0 10.124.176.1
# 6 10.124.208.1 255.255.240.0 10.124.208.1
# 7 10.124.226.1 255.255.255.0 10.124.226.1
#
#
# You’ll need to change the DHCP server IP address in the last command
#
###################################################################
###################################################################

BEGIN { print “Nortel ERS 8600 VRRP to RSMLT Script” }

{ print “config vlan “$1” ip delete “$2 }
{ print “config vlan “$1” ip create “$4″/”$3 }
{ print “config vlan “$1″ ip ospf interface-type passive” }
{ print “config vlan “$1″ ip ospf enable” }
{ print “config vlan “$1″ ip rsmlt enable” }
{ print “config vlan “$1″ ip rsmlt holdup-timer 9999” }
{ print “config vlan “$1″ ip dhcp-relay enable” }
{ print “config vlan “$1″ ip dhcp-relay create-fwd-path server 10.124.16.37” }
{ print “config vlan “$1″ ip dhcp-relay enable-fwd-path server 10.124.16.37″ }

END { print ” – DONE -” }

Let me just step you through the commands. You’ll need to delete the current IP interface, and create the new IP interface, enable OSPF, enable RSMLT, and create the DHCP forwarding agent. If you have PIM or IGMP enabled you’ll need to add a few commands to the list.

Good Luck!

We are looking to move from VRRP to RSMLT during this summer ( 4 Core 8610, 20 distribution switches 8610 and 250 470/5520/BPS switches). Is there a document I can use for the step by step migration?

Thank you,

I had thought I responded but I guess I didn’t (getting old) so sorry about that.

You’ll need to have both ports (A – 1/1 and B – 1/1) in the same Layer 2 VLAN and you’ll need to setup both ports as a S-SMLT (Single port SMLT). You will also need that same VLAN configured on the IST between the two ERS5600 switches. Once you’ve done that you can leave the server setup to “Static Link Aggregation”. I believe you could also use LACP 802.3ad if you wished but in this case a simple single port trunk will work just fine.

That configuration should give you rock solid results. You could also setup VRRP on both ERS5600 switches and point your server to the virtual IP address as the default gateway. If you went that route you would have a solution that is completely redundant and provides for both links in an active/active (1000Mbps * 2 = 2000Mbps * Full Duplex = 4000Mbps) configuration as opposed to using NIC fault tolerant or STP configuration where one NIC is idle waiting to come alive.

Hopefully I’ve explained that well enough.

Good Luck!