Comments on: When it rains it pours!

By: Tom

Tom — Wed, 09 Dec 2009 19:48:16 +0000

We use 4.1.8.3 on our ERS 8600′s and run ~60 VLANs, IST, SMLT, PIM-SM, RIP, SYSLOG, etc. Things have been fairly stable, but it’s because of reported problems as discussed in this thread that we have held off on upgrading to 5.x.x.x. Ever since last year when we found the VLACP timeout issue after installing an ERS 45xx software update (yes, we found that problem and it was a pain) – see http://www.michaelfmcnamara.com/files/VLACP%20timeout%20issue.pdf – I’ve been very careful and leery about software upgrades on core switches. We will continue that policy until I feel more confident that the Nortel ERS 8600 software releases are stable.

About the only problem we have had is some sort of bug with DHCP Guard which caused no end of problems. The solution was to disable DHCP Guard, but long-term, we need to figure this problem out.

But not all is lost. The latest ESM software release is very nice and fixed some nagging bugs. CS1000 6.0 looks very solid, and our SRGs are running with no problems at all. We are very committed to Nortel over the next 5-7 years, but we will continue to move with deliberation when we touch any of the core switches.

By: Mark

Mark — Tue, 08 Dec 2009 22:16:01 +0000

Chuck

We were not experiencing actual LANE lockups on 81612XLRS with 5.0.1. The group of four 10gig ports in LANE one would stop transmitting packets, but was still receiving them. Links up. No errors. We were lucking VLACP logically brought the affected ports down (although they were physically still up) and the SMLT peer took over. Without VLACP the traffic would have been black holed.

By: Mark

Mark — Tue, 08 Dec 2009 21:58:48 +0000

bylie on December 4, 2009 – 2:26 am

Are you running SLPP. In 5.0.1 with SLPP enabled the cpu buffer (99%) filled up on one side of the cluster causing the peer CPU to spike at 100%. This caused instability in the IST. If you globally disable SLPP, the problem goes away. This was due to the SLPP MAC address being placed in the forwarding table twice.

We tested that this is fixed in 5.1.1.1 and 5.0.5

By: Mark

Mark — Tue, 08 Dec 2009 21:53:44 +0000

“ARP issue on a ERS 8600 switch cluster running 4.1.8.0″

Was fixed in 4.1.8.3 How can you manage so many versions of code? We’re always on the same code base with clusters and even stand alones now.

By: Mark

Mark — Tue, 08 Dec 2009 21:50:44 +0000

We only use VRRP when we have to. We use RSMLT which helps when SMLT converges when an 8600 in the cluster is coming back up. The problem is you can only have 32 SMLTs per RSMLT instance. We found this out the hard way. If you go over 32 it cause instability in the IST and things go nuts. A couple of VLAN (like the managment one) have way more than 32 SMLT instances.

We do have unique VRRP IDS across the switch.

By: Chuck

Chuck — Tue, 08 Dec 2009 17:01:00 +0000

We’ve been a Nortel customer for over a decade and have used the 8600 since its initial release. We’ve also beta tested just about every single release since then and have been through more than our share of problems. The SMLT rearchitecture of 4.1.8, 5.0.1 and 5.1 has been a rough ride. 5.1.1.1 finally fixes most of our issues with IST and box instability. We are still working on one more issue that seems to be related to HA-CPU & PIM/IP Multicast that causes the IST to go down when disconnecting an SMLT edge link. With PIM disabled, the issue disappears. We also have an 8612XLRS in use on 5.1.1.1 and have not experienced any LANE lockup issue.

By: SilverMachine

SilverMachine — Tue, 08 Dec 2009 12:32:47 +0000

Hi Michael,

No, they are not runnig 5.1.1.1 code with 8612XLRS modules. No LANE lock-ups until now. They are running 5.1.1.1 code for more than 2 weeks without problems.

Concerning DHCP Relay Agents on a cluster running 5.1.1.1 code: This IP should not be the same as configured for Virtual IP Address (VRRP). It will start a broadcast storm (DHCP Offers packets) on the IST link!!

Best Regards!

By: Michael McNamara

Michael McNamara — Tue, 08 Dec 2009 03:07:49 +0000

Hi Mark,

I’m happy that another Nortel customer is running 5.1.1.1 software without any issues. Also happy to hear that you haven’t experienced any LANE lockups on 5.1.1.1 software.

That’s an interesting comment regarding the SuperMezz cards and SLPP. Do you use VRRP in your configuration? Do you have unique VRRP IDs across the switch cluster? I ask because there is a known issue that we discovered last May that may be related. In short you need to have unique VRRP IDs across the switch, you can’t use VRRP ID 1 for VLAN 1 and VRRP ID 1 for VLAN 2, VRRP ID 1 for VLAN 3, etc.

Thanks for the comment!

By: Ben

Ben — Tue, 08 Dec 2009 02:58:59 +0000

The Cisco VSS solution has worked excellently in our environment. Unlike the ERS8600 it operates 100% as one logical switch similar to a stack of Nortel 5510s or Cisco 3750s. We have redundant fiber to all of our closets and have them setup as Machine EtherChannels (MECs) which is similar to SMLT, but in my experience works a lot better. There is only one active supervisor between the two chassis and the active/active detection is done both by the VSS (SMLT) links as well as by a dedicated heartbeat link.

By: Michael McNamara

Michael McNamara — Tue, 08 Dec 2009 02:56:55 +0000

Hi Udo,

Thanks for the comment. It would seem that a lot of folks are happy with 5.1.1.1 software, however, I will warn you that I’ve heard about issues with the 8612XLRS modules. You might want to contact Nortel support to inquire about any known issues with the 8612XLRS modules if you haven’t already installed them.

Cheers!