22nd July 2008

Ethernet Switch 470 Stack Troubleshooting

posted in EthernetSwitch | 1,246 views | Print This Post

I’ve pulled more than a few hairs from my head troubleshooting stack (cascade) link issues when stacking multiple Nortel Ethernet Switch 460 or Ethernet Switch 470 switches together. I thought I would try to throw together a quick process for testing the cascade module and cable. I hope to make a follow-up post covering the Ethernet Routing Switch 5500 series at a later time.

Let me describe a typical scenario and then offer some ways of isolating the potential problem. You have a stack of four ES470s we’ll refer to as Unit 1, Unit 2, Unit 3 and Unit 4. ES470CascadeCablesWe can use the picture to the left to visualize what a stack of four Ethernet Switch 470s might look like. While all the Up/Down stack lights should be green let’s just say that Unit 3 Down and Unit 4 Up is amber.

Let me just warn you that I have yet to figure how to truly identify a bad cascade module (the module that is built into the switch) from a bad cable without using either a cascade module that is know to be good and/or cascade cable that is know to be good in a process of elimination.

How you can determine if you have a bad cascade cable or cascade module?

It’s really pretty easy although it will require you to take the switch down and use the diagnostic boot code. You’ll need to cable up to the serial interface of the switch in order to run the test. When you’re ready go ahead and cold boot the switch. When you see the following, “470-24T  Diagnostics 3.6.0.7″ (or something similar since you may not have a 24T but a 48T) you’ll need to interrupt the boot sequence by hitting Ctrl-C (go ahead and hit it repeatedly). You should see something similar to the following;

470-24T  Diagnostics 3.6.0.7

Testing main memory - PASSED

>> Break Recognized - Wait..

>> Break Recognized - Wait..

Press 'a'  to run Agent code
Press 'c'  to run Cascade external loopback test
Press 'd'  to Download agent code
Press 'e'  to display Errors
Press 'i'  to Initialize config/log flash
Press 'p'  to run POST tests
Press 'r'  to Receive cascade test packets
Press 's'  to Send    cascade test packets..

Once your at this point you’ll need to take a single cascade cable and loop it between the Up and Down port of the switch your working on. This will put a physical loop between the two interfaces so we can run and external loopback test across the cascade links. When you’re ready go ahead and select “c” from the diagnostics menu.

Test 501  Stack External Loopback    -        FAILED
NSX SXLB STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7

In my case the Ethernet Switch 470 24 Port switch that I was using failed the loopback test. I then took a cascade cable that I knew to be working and repeated the test. It subsequently failed again which indicates to me that the cascade module is faulty. If you were to select “e” from the diagnostics menu you might seem something similar to the following;

System Resets  =       58.

Burn-In Loops  =        0.
Burn-In Errors =        0.
Auto-Burn-In   = DISABLED
Diag Baud      =     9600.

Error Log:
Bad Port Mask  = 80000000
Loop Test Error Description:
  50  501 STAK: Stack Secondary Rx (1) Timed Out
  50  501 STAK: Stack Upstream Clock Failed. Is Cascade Cable Missing?
  50  501 STAK: Stack Secondary Rx (1) Timed Out
  50  501 STAK: Stack Secondary Rx (1) Timed Out
  50  501 STAK: Force Stack RNGO Low Failed Test=0 GCReg=60
  50  501 STAK: Force Stack RNGO Low Failed Test=0 GCReg=60
  56  501 STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7
  56  501 STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7
  58  501 STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7

One very important note! You can only stack switches that are running the same version of software (boot code and agent code). I believe the “Base” light will blink amber if you try to stack two switches together that are not running the same software.

You can also confirm a cascade/stacking issue remotely using Nortel’s Device Manager. Here’s a screenshot of two Ethernet Switch 470s stack together. You can see the yellow LEDs on Unit 1 Up and Unit 2 Down.

Device Manager

Device Manager

I will let you know that we’ve had our own share of cascade modules go bad over the past five years. While the cascade modules appear to be “replaceable” they really not designed to be field serviceable. If a switch fails the cascade loopback test it’s really only good for stand alone operation.

Cheers!

This entry was posted on Tuesday, July 22nd, 2008 at 10:00 pm and is filed under EthernetSwitch. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

There are currently 6 responses to “Ethernet Switch 470 Stack Troubleshooting”

Why not let us know what you think by adding your own comment! Your opinion is as valid as anyone elses, so come on... let us know what you think.

  1. 1 On July 23rd, 2008, LGonsalves said:

    One other way to check the status of each stack switch is to telnet the stack IP and choose menu option Display Hardware Units. This way one is able to see what’s going on with the stack. If a cascade module is faulty, the corresponding switch won’t appear. Also check the
    numbering of the units as it gives clues on which unit of the stack may be faulty or erroneous.

    Normally, after building a stack I usually check for hardware problems using the Display Hardware option. If something seems to be wrong, then the procedure you referred is followed for each switch individually.

    Yes, you’re right: you stack two units with different software and the amber light shows up :)

  2. 2 On July 23rd, 2008, Michael McNamara said:

    Thanks for that additional tip LGonsalves!

    I have seen many occasions where only one port on the cascade module was defective. In that scenario I don’t believe the Ethernet Switch 470 menu or CLI will show you the status of the cascade interfaces. Here’s the “Display Hardware Units” menu for the same switch pictured above in Device Manager.

                             Hardware Unit Information
    
    Unit# Switch Model     MDA Model   GBIC Port   GBIC Port   Cascade SW Version
                                                               Module
    ----- ---------------- ----------- ----------- ----------- ------- ----------
      1   470-48T                      (47) SX     (48) SX     400-ST1 v3.7.1.08
      2   470-48T                      (47) None   (48) None   400-ST1 v3.7.1.08
    

    You either need to visual inspect the switch or query it using Device Manager.

    Thanks for the comment!

  3. 3 On July 30th, 2008, Tom said:

    Good point on the software being at the same rev. We always have to update the software on replacement switches received from Nortel because we run the secure runtime image.

  4. 4 On July 30th, 2008, Michael McNamara said:

    It’s quite a pain when your rushing to replace a dead switch but it’s a show stopper if you arrive at the remote site with a switch that won’t join the stack because it’s running the wrong software version.

    Thankfully Nortel has addressed this problem in the Ethernet Routing Switch 5500 series. You can stack switches with different software versions (I think you may need to be running at least software v4.2 or later) and the switches will upgrade/downgrade to the version running on the base unit.

    Thanks for the comment!

  5. 5 On September 10th, 2008, Munika said:

    hi,
    i am sorry i am posting my comment here. our company has recently got 5 nortel baystack 460 switch. they were being used by other company. when i power up the switch it gives me a command prompt and i figured out through internet it is suppose to give me a menu which my switch doesnt give me.it asks me for a password. (i dont have the password) , tried to break the boot sequence and then it said nvram formatted and all but it still asks me for password for menu.i figured out through command line how to enter ip and configure. once i bring the switch on my network through command line then i tried to use the web interface. i did see the web interface but it just shows system configuration and shows “access(RW)” i think i should be able to configure my switch through web interface. any idea how can i get those configuration settings so that i can configure through web interface. i got similar 5 switches and i will have to cascade them too. any help is appreciated. thanks.

  6. 6 On September 10th, 2008, Michael McNamara said:

    Hi Munika,

    What version of software is the Ethernet Switch 460 running? If you followed the procedure outlined at http://blog.michaelfmcnamara.com/2007/11/factory-reset-nortel-ethernet-switch/ the switch shouldn’t prompt you for either a username or password (although it might if you are running software 3.7 or later). If the switch is running the SSH version of the software there won’t be any menu available just the basic CLI interface (there isn’t enough memory for both the SSH software and the menu).

    The default read-write username is RW with a password of secure.
    The default read-only username is RO with a password of user.

    I would suggest you register for an account on the Nortel website and download the documentation for the version of software the switch is running. Depending on the version of software you should be able to configure the vast “majority” of options through the web interface although there will be some that must be configured through the CLI interface and/or Nortel’s Java Device Manager (JDM).

    Good Luck!
    Mike

Leave a Reply

  • Polls

  • What vendor would you use for Enterprise LAN/WAN switching if Nortel disappeared tomorrow?

    View Results

    Loading ... Loading ...
  • Archives