We recently started deploying RSMLT in place of VRRP and have run into very few problems. With that said there is one option that is disabled by default that is a must for anyone running in an RSMLT configuration. Since the peer IP addresses are learned dynamically a problem can occur if both ERS 8600 switches go down but only one of them returns to service. In my environment we’ve been using .1 and .2 for our ERS8600 interfaces with DHCP serving up .1 as the default gateway. If the .1 ERS8600 never recovers the .2 ERS8600 will never be able to learn the IP peer interface and hence will not be able to answer requests for .1 from the DHCP clients.
I would advise anyone using RSMLT to enable rsmlt-edge-support with the following command; config ip rsmlt rsmlt-edge-support enable. You will need to be running software release 4.1.4 or later.
RSMLT implementation does not use a Virtual IP address but instead uses Physical IP addresses for redundancy. At the same time, RSMLT can be deployed in either L3/routed configurations, or L2/edge configurations, where previously one might have used VRRP (and back-up master). Now previously, if there was a power outage or shutdown of both switches within a Dual Core IST pair and for some reason only one switch came back-up, clients using the powered-off switch’s IP/MAC as their default gateway would lose connectivity to the network. In such a scenario, even through RSMLT is enabled on the switch it was unable to backup for the Peer as it was unaware of the Peer’s IP/MAC address. With this new feature, the RSMLT Peer’s IP and MAC addresses are stored in the config file and will be used upon next reboot, if the IST link does not become active and operational. Otherwise, the switches will learn from their Peer as normal; see below for additional information.
This feature can be enabled/disabled by the following CLI command:
config ip rsmlt rsmlt-edge-support <enable/disable>
When the configuration file is saved, if the rsmlt-edge-support flag is enabled and RSMLT peer is UP, the peer IP address and MAC address is also automatically saved.
The peer information is cleared by the following CLI command.
config ip rsmlt clear-rsmlt-peer [<vlanId>]
NOTE: if the peer information is cleared the switch could stop forwarding for the peer.
After both the Dual Core IST switches have come back-up, and if the IST comes up and is operational, if a RSMLT Peer enabled message is received from the peer, then normal RSMLT operation is followed. If the peer has either an IP or MAC change, then a new save config must be performed in order for the new information to be saved, and RSMLT L2 Edge support to operate correctly. However, if the IST Peer up message is not received (for example RSMLT is not enabled properly) and the rsmltedge- forward flag is enabled, then first the RSMLT Hold down timer is started to allow routing protocols to converge; during this time user operation could be affected. Once the hold down timer expires, saved peer information is picked up and Switch starts to backup for the Peer by adding the previously saved MAC and ARP records. The Holdup timer is then started and once this timer expires the previously added MAC and ARP records are deleted and the switch stops backing up for the Peer, as the Peer is not running proper RSMLT for the VLAN. It should be noted that RSMLT is a per VLAN parameter, and therefore all affects are on a per VLAN basis, not necessarily per switch. In L2 Edge support mode the Local values of the HoldDown timer (default value of 60 seconds) and HoldUp timer (default value of 180 seconds) will be used.
Cheers!
Hello,
What was the driver for the move from VRRP to RSMLT? We’ve looked at it here, but with only ~1200-1500 network end nodes, I was not sure it made sense for a site as small as ours.
Hi Tom,
I’ve had some very long discussions with Nortel and they feel that RSMLT is much more efficient and scalable than VRRP.
I had the opportunity this week to take a single core ERS 8600 and make it into a cluster of two ERS 8600 switches so I used VRRP for VLANs where multiple ERS 8600s (closet switches) were congregating and I used RSMLT for VLANs that just had Layer 2 edge switches such as Ethernet Switch 470s and Ethernet Routing Switch 5500 series switches.
It really has more to-do with scaling than anything else. If you only have less then 100 VLANs then you are probably fine but if you start to move past that point it makes more sense to move to RSMLT to be more efficient with the CPU utilization.
I’m happy to report that so far RSMLT seems to be working fine… although you most certainly need to set the hold-up timer to infinity (9999) and enable rsmlt-edge support.
Thanks for the comment!
hello,
I have 2 ERS8600 to connect one switch Cisco,
I have trial using existing configuration in EX. ERS_8600_0 and ERS_8600_1 using LACP configuration to connect to L2 SW.
But in this trial the VRRP in ERS_8600 is not UP, I can ping IP physical both ERS_8600 from our laptop but not IP VRRP.
Can you help us to analyst this configuration so I can ping IP VRRP.
Thanks
Hi Fredi,
I’m going to assume that you have the IST/SMLT configuration setup properly on both core ERS 8600 switches and you have an 802.3ad configuration on the Cisco edge switch.
There is an option that can be set to ignore ICMP requests to the VRRP address. If you want to allow ICMP requests to the VRRP address then you should issue the following command;
config ip vrrp ping-virtual-address enable
This assumes that the laptop in question is actually able to communicate with the network, the only problem is that it can’t ping the VRRP interface.
Make sure you have disabled Spanning Tree on the ERS 8600 uplinks to the edge and on the downlinks from the Cisco edge switch.
Good Luck!
Hello,
I was wondering whether you are familiar with SMLT/MLT to servers. 2x ERS5600-switches with IST running between them. I am trying to come up with a scenario whereby Port 1 on Switch A and Port 1 on Switch B goes to a single server. On the server side, I have different NIC teaming options (Adapter Fault Tolerance, Adapter Load Balancing, Static Link Aggregation, Switch Fault Tolerance, 802.3ad Dynamic Link Aggregation). I tried numerous options but I am getting very inconsistent results.
Here is a typical setup for the server/switch
Server: 10.0.0.3/24
Switch A: 10.0.0.1/24
Switch B: 10.0.0.2/24
The server can use either 10.0.0.1 or 10.0.0.2 as the gateway. For testing purposes, I am using 10.0.0.1
I ran 3 constant ping tests to Switch A, Switch B, and 4.2.2.2 (DNS server).
These ERS5600 switch cluster is part of an RSMLT domain. I am able to Ping out to 4.2.2.2 independently from each switch. I am not sure what would be the setting on the switches to get high resiliency working to the server. Do you have any configuration ideas?
The problem I am having is that it seems that I was able to PING both switches when using some of the teaming options with no configuration needed on the switch at all. I was however unable to ping to 4.2.2.2 unless I connect back all the Ethernet links. It seems that is is favoring one switch over another despite being able to ping both gateways.
Hi Nicolas,
I had thought I responded but I guess I didn’t (getting old) so sorry about that.
You’ll need to have both ports (A – 1/1 and B – 1/1) in the same Layer 2 VLAN and you’ll need to setup both ports as a S-SMLT (Single port SMLT). You will also need that same VLAN configured on the IST between the two ERS5600 switches. Once you’ve done that you can leave the server setup to “Static Link Aggregation”. I believe you could also use LACP 802.3ad if you wished but in this case a simple single port trunk will work just fine.
That configuration should give you rock solid results. You could also setup VRRP on both ERS5600 switches and point your server to the virtual IP address as the default gateway. If you went that route you would have a solution that is completely redundant and provides for both links in an active/active (1000Mbps * 2 = 2000Mbps * Full Duplex = 4000Mbps) configuration as opposed to using NIC fault tolerant or STP configuration where one NIC is idle waiting to come alive.
Hopefully I’ve explained that well enough.
Good Luck!
Hello Michael,
We are looking to move from VRRP to RSMLT during this summer ( 4 Core 8610, 20 distribution switches 8610 and 250 470/5520/BPS switches). Is there a document I can use for the step by step migration?
Thank you,
Hi Adrian,
It wasn’t very hard at all… I actually took some time to write an AWK script to help remove any “fat-finger” issues.
Let me dig up that script right now… even if you don’t care to use the script it will give you a good idea of the steps necessary.
#!/bin/awk -f
#
###################################################################
###################################################################
#
# We need to gather the input data from the CLI interface of the
# switch. You’ll need the VID, OLD IP, NETMASK and NEW IP.
#
# Here’s the data file format that the AWK script expects;
#
# VID OLD IP NETMASK NEWIP
# 1 10.124.20.201 255.255.240.0 10.124.16.1
# 3 10.124.80.1 255.255.240.0 10.124.80.1
# 4 10.124.160.1 255.255.240.0 10.124.160.1
# 5 10.124.176.1 255.255.240.0 10.124.176.1
# 6 10.124.208.1 255.255.240.0 10.124.208.1
# 7 10.124.226.1 255.255.255.0 10.124.226.1
#
#
# You’ll need to change the DHCP server IP address in the last command
#
###################################################################
###################################################################
BEGIN { print “Nortel ERS 8600 VRRP to RSMLT Script” }
{ print “config vlan “$1” ip delete “$2 }
{ print “config vlan “$1” ip create “$4″/”$3 }
{ print “config vlan “$1″ ip ospf interface-type passive” }
{ print “config vlan “$1″ ip ospf enable” }
{ print “config vlan “$1″ ip rsmlt enable” }
{ print “config vlan “$1″ ip rsmlt holdup-timer 9999” }
{ print “config vlan “$1″ ip dhcp-relay enable” }
{ print “config vlan “$1″ ip dhcp-relay create-fwd-path server 10.124.16.37” }
{ print “config vlan “$1″ ip dhcp-relay enable-fwd-path server 10.124.16.37″ }
END { print ” – DONE -” }
Let me just step you through the commands. You’ll need to delete the current IP interface, and create the new IP interface, enable OSPF, enable RSMLT, and create the DHCP forwarding agent. If you have PIM or IGMP enabled you’ll need to add a few commands to the list.
Good Luck!
Hi Michael,
First of all thank you for the script. It will really help. Secondly let me recap and correct me if I am wrong:
1. Disable OSPF
2. Delete old and create new interface
3. Enable OSPF
3. Enable RSMLT at the Core
4. Enable DHCP forwarding
5. I am running IGMP with DVMRP – do I have to disable multicast and reenable after?
Any issues with BPS stacks ?
Adrian
Hi Adrian,
You won’t have any issues with your edge switches but you will have some brief routing/reachability issues. You should expect some brief routing/reachability issues on the affected VLANs while you’re making the changes. Once you’ve restored the IP interface and OSPF has converged you should be good to go.
I’d also recommend that you reboot (yes I said reboot) your ERS 8600 after you’ve made all the changes and everything looks good. I’ve found that a reboot/restart/cpuswitchover really helps provide some stability to the environment after a large number of changes such as those being discussed here.
You also need to make sure you enable RSMLT edge support (documented above) and let’s not forget to enable RSMLT globally and not just on the individual VLANs.
With regard to IGMP and DVMRP you’ll need to check what happens once you delete the IP interface. I haven’t tried it myself but I suspect that without an IP interface IGMP and DVMRP might become disabled as does the OSPF interface and the DHCP relay agent.
Good Luck!
Hi Michael,
I notice this article is about year old and I had already configured RSMLT exactly as described above and it has been working perfectly for Layer3 for some time. I have just this week enabled Multicast PIM-SM on this pair of 8600’s feeding multiple 5530 stacks running DMLT connected to the 8600’s.
The problem I have is with Multicast only – but exactly as described in your post. Layer 3 returns almost immediately but Multicast never recovers on it’s own.
I need to bring back up the (.1) 8600 which is the default gateway, or for “test purposes only – to see the behaviour”, change the (.2) 8600 address to (.1) or add a static route to the 5530’s (the RP’s) via the (.2) 8600.
All of which re-establish proper multicast operation.
The last two are not able to be done in practice – obviously.
At no time is layer 3 affected, it handles the loss flawlessly. It’s just multicast that gets screwed.
Any insights?
Cheers…
Hi Dennett,
You’re probably forgetting to re-enable PIM on that specific VLAN. When you delete the IP interface the switch will remove all protocols that are dependent on that IP interface. That’s why you need to re-enable DHCP and then re-create the DHCP forwarding agent. Likewise if you look at the PIM (PIM-SM) tab for that VLAN you’ll probably find that it’s unchecked (disabled). You’ll need to enable PIM and you might need to recreate some of your PIM configuration such as the RPs. You should probably look at your old configuration (prior to any changes) and then just cut-n-paste the related PIM commands back into the switch.
Good Luck!
Hi Michael,
Thanks for the reply and great site by the way…
I probably should have provided more info in my first post but wasn’t sure this thread would still be active after a year.
There is no problem with the config of either of the clustered 8600’s nor with the 5530’s. PIM remains enabled for all VLAN’s and the RP’s are fine. The entire configuration on all devices by the way is STATIC. Routing, RP’s etc, etc.
The simple fact that getting Mulicast working again on the backup 8600 (.2), is a simple as entering a static L3 route into the 5530’s pointing to (.2) as the gateway to the RP’s, tells me that all of the PIM-SM info is correct and working.
It is just that Multicast does not like it, when (.2 backup) assumes the (.1 IP and MAC) when the master 8600 is taken down. I am doing all of this for testing prior to production implementation. But at the moment it doesn’t look good for Multicast.
Multicast over VRRP has similar (but more varied) failures. RSMLT is supposed to have fixed this.
I’m using 5.1.2 on the 8600’s and 6.2.200 on the 5530’s.
The last one is not GA.
Thanks…
Hi Dennett,
Thanks for the kind comments.
I believe I now understand the problem you are experiencing.
Are the edge switches Layer 2 or Layer 3?
Have you tried your configuration with a dynamic BSR?
Also you could probably tie your RPs to a CLIP interface and not the physical IP interface for any specific VLAN. I did find an interesting note in a post I made a while back regarding the 4.1.8.2 software release; Not all IP Multicast failover or fail-back/recovery situations can provide such times today, as many situations depend on the IPMC protocol recovery. For best IPMC recovery in SMLT/RSMLT designs, the use of static RPs for PIM-SM is recommended, with the same CLIP IP address assigned to both Core IST Peers within the Cluster, and to all switches with a full-mesh or square configuration.
If you look at my post titled,
Michael McNamara says
Dennett says
Michael McNamara says
Dennett says
Dennett says
Dennett says
Michael McNamara says
Dennett says
sayantani sarkar says
Michael McNamara says
sayantani sarkar says
Tuan Phan Anh says
Ramkumar says
Michael McNamara says
Rajesh Bisht says
Michael McNamara says
Here are some additional documents that might help shed some light on the issue;
NN46205-200_02.02_Planning-Engineering_Network-Design.pdf
pp8600_tcg_pim_sm_public.pdf
You’ll find the PIM-SM document is a little old but should still serve as a good resource.
Cheers!
Hi Michael,
Thanks again, I’ll read all of that.
just for clarity the 5530’s are Layer 3. (Distribution switches)
No have not tried dynamic BSR, trying to keep everything static if possible but this may not be practical.
There are also be layer 2 (edge switches) downstream from these, which I will also add to the lab testing.
Production is more complex (wish I could put up a diagram) – 8 x 8600’s full mesh, layer 3 distribution switches (5530’s), layer 2 edge switches (4548’s). (Too many to count) – around 30,000 users.
The 5530’s will be the RP’s for each building (around 55 buildings).
Just need to be sure this is going to work before I commit to production.
Dennett
That’s a nice sized network, I won’t dare to ask how many actual ports if you have 30,000 users.
Just a quick word of warning, if you are running the ERS 5530s in an IST/SMLT cluster configuration then PIM-SM is not supported on those switches, only IGMP. You’ll need another switch/router to run the PIM-SM functionality.
There were several discussions over on the forums that might not be of too much value but you might be interested in reviewing anyways.
Let us know how you make out if you have the time.
Good Luck!
Hi again,
No, there are no 5530’s with IST.
They are all stacked DMLT, back to RSMLT on the core 8600’s.
From the 5530’s downstream – MLT to the 4548’s.
Cheers…
Hi Michael,
I think I may have found the cause of the problem for the multicast failure over RSMLT and perhaps the reason why multicast fails over VRRP.
I will do some more testing in the lab tomorrow and will post my results. They may help someone else.
Thanks for your input and interest …
Cheers…
Dennett
Hi Michael,
I tested things in the lab today and thought I’d let you know and your readers how it went.
A bit of background about the reason Multicast fails over RSMLT and over VRRP when the Master 8600 dies.
The lab test 5530 distribution switches use a default unicast route pointing to 8600A(10.10.10.1) for Layer 3.
When 8600A(10.10.10.1) fails, 8600B(10.10.10.2) assumes responsibility for 8600A’s(IP/MAC) – RSMLT at work.
This is fine for Layer 3 but Multicast routing is a bit more complex.
The multicast routing (Mroute) table is (in my case with STATIC configuration) built from the information in the unicast routing table. Because I only have a single default route, there will only be one entry in my Mroute table.
The important bit of information in this table, is the RPF (Reverse Path Forwarding) neighbor. In my case (10.10.10.1) again.
PIM-SM also keeps a PIM neighbors table on each device, which is populated in the 5530’s case, by PIM Hello messages sent from the 8600’s every 15 secs.
My test 5530 stack, has two PIM neighbor entries, one for 10.10.10.1 and one for 10.10.10.2 (These entries expire after 90 seconds if no Hello message has arrived)
When you have a Multicast Source or Client on your switch, a Join/Prune message is generated by the switch and forwarded to each RPF neighbor in the Mroute table. BUT – “O N L Y” if there is a corresponding entry for that IP in the PIM neighbor table. See the PIM-SM RFC…
When 8600A(10.10.10.1) goes down, it’s record in the PIM neighbor table expires after 90 seconds and disappears. (No more PIM Hello’s…)
Now when the 5530 builds the Join/Prune message, it checks the Mroute table (we have only one entry for 10.10.10.1), which it uses, then looks for 10.10.10.1 in the PIM neighbor table. It is no longer there, so the message is never sent. See RFC again…
There will never be a Join/Prune built for 8600B(10.10.10.2) because there is no Mroute entry for 10.10.10.2 in our table.
Multicast therefor fails.
As an aside, the reason multicast fails on VRRP is similar. The PIM Hellos from the 8600’s come from the physical addresses, not from the VRRP VRIP.
So there will never be an entry in the PIM neighbor table for a virtual IP address.
OK – How Do We Fix This (for STATIC environments)?
We could configure a STATIC Mroute entry for 10.10.10.2 with the CLI command config ip static-mroute create rpf
[preference ]
Now an RPF Neighbor entry for 10.10.10.2 is in our Mroute table and our PIM Neighbor table and a Join/Prune can be built and sent to 10.10.10.2
Multicast now continues if 8600A dies.
UNFORTUNATELY, STATIC Mroute entries are NOT supported by the 5500 series switches, so –
We configure a virtual PIM Neighbor entry for 10.10.10.1 with the CLI command ip pim virtual-neighbor 10.10.10.3 10.10.10.1
This is done in the 5530 (10.10.10.3).
Now there will always be an entry for 10.10.10.1 in the PIM Neighbor Table and the Join/Prune message can be successfully sent. (10.10.10.1 is in our Mroute table and in our VirtPIM Neighbor table).
Multicast now continues if 8600A dies.
For VRRP if your VRIP was 10.10.10.4 you would ip pim virtual-neighbor 10.10.10.3 10.10.10.4
I will be testing ECMP as an alternative next week and update if you are interested.
I hope this helps someone. And hope I’ve made no typos. It was an interesting problem and fun to debug…
I could do I diagram and email it to you to include, Michael. Just let me know.
Cheers…
Dennett
Thanks for the update Dennett.
It’s been a while since I’ve worked with a PIM-SM Multicast configuration so some of the details are a little fuzzy but I can follow your logic.
Would love to hear the ultimate outcome. I might try to set this up in my lab just to confirm but my time is very tight these days.
Any thoughts about making this into a guest post? How to setup PIM-SM on and ERS 8600 switch cluster for HA?
Thanks for sharing!
Hi Michael,
I’ve done a simple Visio diagram, that makes the descriptive stuff above, more meaningful.
I’m happy to email it to you if you’d like to put it up.
Cheers…
Dennett
Hi Michael,
i have VRRP configured in my netwotk with back-up master enabled for 8600. do i need to disable it prior configuring RSMLT.
Sayantani
You should either run VRRP or RSMLT but not both at the same time.
It’s not a simple check box… you most likely need to re-address your Layer 3 (IP) interface.
See comment #8 above for a script that I used to make the change.
Cheers!
ya…i ll run RSMLT only. but your commant “re-addressing of layer 3 interface”…could you pls explain wat does mean..
Dear Michael,
Thank you very much for this document, it really very very excellent
Hey Michael ,
Your document was very crisp and useful but in case of edge deployment can we leave the holdup timer by default (180) , because when Core A is completely down , core B will forward the traffic destined for Core A only for 180 seconds right ? I read some documents saying it should be 9999 (infinite ) isn’t it ?
Regards,
Ram
Hi Ramkumar,
You are correct… you want to set the holdup timer to 9999.
Cheers!
Hi Michael,
I am a bit new to Avaya Data platform, so need your guidance for a proposed network changes in our setup. I am not sure if this is the right forum for that.
We have two different networks with different set of Cores ( VSP 8400 with RSMLT in one setup & ERS 8800 with RSMLT in second setup ). Now we have a requirement of merging these two networks. I know that two switch clusters of same type can be merged, but not sure if switch clusters of two different models can be merged or not. If you could confirm that , it will help.
Also, this is our requirement after the merging. please advise if these can be achieved by merging two clusters.
1. VLANs of Cluster 1 should be able to access some VLANs of cluster 2 & vice versa. Do we need to create same set of VLANs on both clusters? or routing them using RSMLT port configuration will work.
2. Static routes configured on both the clusters should still be able to work as before. Is this workable?
Please advise.
Regards,
Rajesh
You can certainly “merge” the VLANs between the two solutions… assuming this is a temporary solution until you remove one or the other you can just setup a simple SMLT trunk between the two. I would just set it up as a simple SMLT configuration on each side, remember the two clusters have no idea what they are connecting to so from that perspective it’s just a simple LACP over SMLT configuration.
Cheers!