Hi Milko,

I can’t point you to a link other than the Nortel website.

The RS modules are just newer modules offering additional density and port combinations. In all cases the the RS modules are oversubscribed, something that Nortel generally doesn’t do but Cisco is well known for.

The slots on the ERS 8600 only provide 30Gbps of switching fabric so if you have an 8630GBR module with 30 1Gbps SFPs then there is no over subscription. If you take an 8648GBRS which has 48 1Gbps SFPs then there is some amount of over subscription but not much. Finally take an 8612XLRS which has 12 10Gbps XFPs then there is about 4:1 over subscription.

Hopefully that answers your question.

Feel free to join us in the forums; http://forums.networkinfrastructure.info/

Cheers!

Hi Vince,

You should see a file on the PCMCIA card that ends in either *5.000 or *6.000 depending on which slot the CPU/SF is in. That is the logfile, now if you didn’t have a PCMCIA card inserted that would be a problem. When you list out the filesystem contents what do you see?

Here’s an example from my testlab switch;

sw-8600a-testlab:5# dir /pcmcia
/pcmcia/295c0005.num
/pcmcia/70700005.000
/pcmcia/70700005CriticalLogFile.000

Again if you want please start a thread over in the forums and we can discuss further.

Good Luck!

I checked the PCMCIA card and there are no log files present.

The option logtoPCMCIA is also set to true.

Hi Vince,

I have seen several occasions where some of the R and RS blades/cards will reset themselves and the configuration for that slot could default itself. Actually I’ve seen it more frequently on non-R and non-RS blades/cards when you replace defective blades/cards. If you hot swap a 8648TXE blade you may find that the configuration for that slot can default itself after you place the new card into the chassis. The quick solution to this problem is to reset the switch which will cause it to load the saved configuration. A few folks might suggest using the “source” command from the CLI interface but I’ve had issues using that command and wouldn’t suggest you use it without first testing.

In your specific case it sounds like you had two problems…. 1) something caused the 8612XLRS to go offline and 2) someone accidentally saved the configuration of the switch with the card offline essentially removing the configuration for that slot. I have seen quite a few R and RS blades/card reset themselves but the configuration issue has always been tracked back to human error, someone accidentally saving the configuration while the card is offline.

If you maintain backups you can probably easily restore a previous version of /flash/config.cfg which should resolve your configuration issues, although I would advise you reboot the switch to load the restored configuration. As for how to protect yourself for a repeat event in the future… there are a number of configurations/features available that could have helped prevent your network from collapsing. You can protect the network from a 8612XLRS (or any card for that matter) losing it’s configuration but coming back online creating a physical loop. Don’t use the Default VLAN (VLAN 1), enable tagging (Perform Tagging) on your trunks, enable Discard Untagged Frames, and use VLACP between your core and edge switches. In a default configuration VLACP won’t be enabled and the edge switch would ignore the uplink that connected to the now miss-configured core port preventing any loop from forming in the first place.

As for what caused the initial problem you’ll need to examine the log files that will be stored on the PCMCIA card.

If you post over on the forums, attaching the log files, I might be able to help you analyze the log files.
http://forums.networkinfrastructure.info/nortel-ethernet-switching/

Good Luck!

We have two Nortel 8600 Cores with the following cards:
– 1 x 8692 CPU
– 4 x 8683XLR
– 1 x 8612XLRS

Our edge devices are connected with 10GB SMLT to Nortel 5500/5600 stacks.
Yesterday our network went down and we noticed that on Core A the 8612XLRS card had an amber online light.
Core A was shutdown and that point users where able to access the network.
I brought Core A back online however before I did that I unplugged all the edge connections.
The amber light on the 8612XLRS was now green.
I noticed that the configurations on Core A for SMLT had all disappeared.
I am assuming that something happened to the card and it lost its configuration however the 8600 did not let the other 8600 know and therefore a loop may have been caused on the network.
Have you ever seen 8612XLRS cards appear orange and then losing their config?
Even if the card had an issue and lost its configuration shouldn’t Core A inform Core B about the issue and not bring the network down?
Nortel has examined our configurations and said that they follow all best practice methods and they could not pin point any issues.
Any input would be helpful.
Thank You.
Vince

Out of curiosity, did anyone end up on 5.0.1.X or greater running in cluster? I’m running around 24 x 8600’s (all 8692 SFs w non-super mezz, some HA/some not) of them on 4.1.8.2 which is proving some stability after a somewhat rough 12 months.

If so, what’s been the experience? and why did you go up, what features pushed you there?

regs,
Stuart

Thanks for your replay. I’ve just saw it too on the 5.0.1.0 release note…

Regards,

Mary

The issues with HA have been resolved in 5.0.1.0 software release.

Cheers!

Regardin the error messages you had in the logs, you stated that they disappeared when you deactivated HA. But what did Nortel say about this ?
Is this a known issue ?

Thanks for your feedback.

Regards,

Mary

It’s my understanding that software release 5.0.1.0 is suppose to be released by the end of February 2009. I’ll be testing 5.0.1.0 software at one of my facilities so I’ll be sure to fill everyone in on my experiences.

I can’t blame Nortel too much for not announcing problems, you don’t see Cisco or HP out there telling you there hardware/software doesn’t work. However, your sales engineer should be able to help steer you in the right direction.

Thanks for the feedback!

now, Nortel seems well aware of the HA issues (even if NO IST cluster/SMLTs are configured; just an access switch running HA) in 5.0.0.1

i now have to schedule another downtime window for my data centers / server access to disable HA from our 8600s.

i’ve spoken with my local engineers and hope to get a response.

has anyone seen publications/notices for 5.0.0.x issues? i am subscribed on all of my equipment/models, but i have yet to see any warnings or notices. if this information would have been made readily available to me, i could have used my downtime / upgrade window to disable HA. now i have to go through a window process again. quite frustrating…

i have gone to 5.0.0.1 on two of my ERS-8600s this past weekend because we have -RS modules that needed to be installed (ASAP).
i have run into the same issue as you regarding HA failure.

we are NOT doing any IST cluster/SMLT on these 2x 8600s…

CPU5 [02/10/09 12:47:56] SNMP INFO HA-CPU: No peer connection is established.
CPU5 [02/10/09 12:43:00] SNMP INFO Cannot communicate with backup CPU
CPU5 [02/10/09 12:42:56] SNMP INFO HA-CPU: Lost connection to Standby CPU.
CPU5 [02/10/09 12:42:56] HW ERROR Lost connection to standby state = 0 event = 0
CPU5 [02/10/09 12:42:56] HW WARNING framework_restart: lost connection to Standby CPU, heartbeat_ticks = -20

this happened on one core about 7-8hours after the 5.0.0.1 upgrade (secondary CPU lost HA sync, and went down HARD). module had to be physically removed and re-inserted into the backplane. it appeared to boot OK, but failed HA sync again when fully initialized. i setup an RMA for this unit, received a new one, and it booted/sync’d with HA no problem. however, a few hours later, it failed again and had to be physically removed/re-inserted.

this was only happening on 1 of 2 8600s that i upgraded to 5.0.0.1…..but the 2nd just now took a hit and HA / secondary CPU is now done on that chassis as well.

—-

case has been opened and escalated with Nortel, and i haven’t heard any word as of yet on a fix…looks like we will have to be disabling HA on our server distribution frames, which will result in a slight service interruption as the master will have to be reset for the disabling of HA to take effect.

what did Nortel inform you as to the problem when you opened your case?

thanks,

our square-full-mesh 8600s will continue to run 4.1 code for some time (eventually, 4.1.8.2)..because we still run 8691s and do not have any of the -R/RS modules installed there. there is nothing in the setup that requires the upgrade.

however,
our server-core/access 8600s have 8692s, and we recently purchased the 8648-GTRS modules…and are now waiting to go to 5.0.0.1 so we can run the -RS’. i don’t think we’ll see any issues, as the server-access 8600s are not doing routing, IST, or SMLT or anything too complicated with that regards. was just curious to see if anyone had any immediate issues with the new point release code.

i have been waiting for a stable code-release to run HA on the core 8600s (doing IST/SMLT/routing), but since that was pulled back in the 3.7.x days, i have been fairly reluctant to even address or enable the HA issue there. don’t think i will attempt it anytime soon even if it is stable in 5.0.1. haven’t had any failures in the past 5years or so where HA would have saved us, and i’d rather have the stability of non-HA right now, then to worry about minor bugs with HA enabled.

thanks again for your replies!

All the changes incorporated into 4.1.8.2 will be incorporated into 5.0.1 so unless you absolutely need to move I would suggest waiting for 5.0.1.

I believe they also plan to have the HA issues resolved in 5.0.1 which has been a high priority issue with the FDB/ARP/IST/SMLT issues.

I stopped using the 8600 for edge access a long time ago and started using the 5500 and 4500 series in stacks. I found the price points to be much easier on the pocket and it provided 10/100/1000 while we all waited for the “Power Ranger” 10/100/1000 card which eventually became the 8648GTR.

Thanks again for the comment!

we have 2 ERS-8600 Server Dist. Frame switches w/ dual-8692SFs that are NOT doing routing (performing as server-access-cores), and need to go to 5.x for the 8648-RS module. time is running out to get this module installed and i will probably move towards 5.0.1 in a week or so.

they are not doing IST/SMLT as of yet. just stand-alone SMLT back to the 8600 cluster (currently and will still be running 4.1.7.2).

hopefully 5.0.1 works well; the configuration on the server cores is quite simple.

if anyone has had any issues, i’d be interested to hear.

I have tested few days ago on 5 ERS8610 (version 5.0.0.1). We have 4 ERS8610, 2 ISt clusters full-mesh SMLT and 1 ERS connected (smlt) to one cluster. We are using HA-CPU in every ERS.
We have some problems with the standby CPU. In the logs we have the following error messages:
CPU5 [11/03/08 07:33:35] SNMP INFO Communication established with backup CPU
CPU5 [11/03/08 07:33:32] HW INFO HA-CPU: Table Sync Completed on Secondary CPU
CPU5 [11/03/08 07:33:29] SNMP INFO HA-CPU: Synchronized state.
CPU5 [11/03/08 07:33:29] HW INFO HA Table sync took 283333 usecs
CPU5 [11/03/08 07:33:29] HW INFO VRF name: GlobalRouter (VRF id 0): HA-CPU: Table Sync in progress
Warning: Please do not reset either of the CPU/SSF cards until table synchronization is fully completed

CPU5 [11/03/08 07:33:28] SNMP INFO HA-CPU: Peer connection is established.
CPU5 [11/03/08 07:30:52] SNMP INFO HA-CPU: No peer connection is established.
CPU5 [11/03/08 07:25:57] SNMP INFO Cannot communicate with backup CPU
CPU5 [11/03/08 07:25:52] SNMP INFO HA-CPU: Lost connection to Standby CPU.
CPU5 [11/03/08 07:25:52] HW ERROR Lost connection to standby state = 0 event = 0

CPU5 [11/03/08 07:25:52] HW WARNING framework_restart: lost connection to Standby CPU, heartbeat_ticks = -26

I have opened a case. I let you know.

Thanks for the comment!

Have you had a chance to load Ver 5.0 software yet and if any issues with it

I haven’t yet loaded the software but I’ve at least got one switch in the lab ready from a hardware perspective with dual 8692SFs (both with SuperMezz cards), a 8648GTR and a 8630GBR card. I hope to load 5.0 software on that switch in the next few days although the work schedule is currently hellacious.

Thanks for the comment!

Can you update us on how the first iteration of 5.0 software is working out? Probably not doing my 5.0 updates until mid-September (three 8600s to do). Part my upgrade strategy is to get rid of the last non-M/E/R blades and 8690SF cores out of the network. We will also be doing the chassis swaps to get the faster backplanes (not happy about this, but the swapout price is small at this time).

I gave my Nortel sales representative the evil eye when I first heard about the new licensing structure at a product briefing. Thankfully he was fully prepared for my displeasure and informed us of the “grandfathered” licenses that would be made available to all legacy 8600 switches.

I just recently upgraded my testlab (two ERS8600 switches both with dual 8690SFs) to v4.1.6.3. Unfortunately the testlab hardware I currently have will not run 5.0 software so I’ll need to scrounge up some spare 8691SFs before I can try loading 5.0 software in the lab. I’m currently planning to blow 4.1.6.3 out to all my major core 8600 switches over the next six weeks.

I would probably advise that you wait for a 5.x release unless you need to added 10GE density offered by the new cards. I can tell you that I’ll probably be waiting.

Thanks for your comment!

Thanks for your post on the “grandfathered” licenses — I wasn’t aware of the program before you mentioned it. I’ve just received an advanced license file for our 8600s, which has brightened my outlook on v5.0 considerably. :)

I just need to decide when/where to deploy it now, but I’m considering waiting for a 5.0.1 or 5.0.2 release rather than taking a chance on 5.0.0.

For those of us looking to deploy 10GB LAN/MANs the new 8612XLRS module is a welcome addition. The current 8683XLR module only provides 3 XFP ports while the new module provides 12 XFP ports. With that said you must also install a cooling module which, takes up another slot in the 8010 chassis. I’m sure the new module is also going to be oversubscribed at the chassis backplane so that would also need to be taken into consideration.

With respect to the licensing, I’ve already had some discussions with Nortel on this subject and I’ve been told that all chassis that have already been purchased by customers will be grandfathered into the program. I’ll talk more about the grandfather licensing in an upcoming post.

I’m actually in the process of removing ERS 8600s from the edge closets in favor of ERS 5500 series switches. Those closet ERS 8600s that remain in place will most likely stay at v4.1.x.

As you mentioned there are a lot of little “annoying” things in the release notes that are sure to haunt some folks as they look to upgrade. Example; Nortel recommends that you reformat your /flash device in the 8691SF/8692SF because they’ve changed some signature in the DOSFS. I believe they also recommend using a minimum 256MB PCMCIA flash card, so much for the 64MB flash card that came with the switch fabric.

Thanks for your comment!