While there were no new features added but there were a number of bug fixes;

• The status of the stack ports are now correctly displayed when interrogating the MIB for the stack (Q02082410)
• If MAC addresses are quickly aged out of the forwarding database when the device is authenticated via Non-EAP memory is no longer memory incorrectly consumed (Q02090742)
• Previously the ifOperStatus reported the status of the IP Management interface based on the port status. Now the switch reports status based on the virtual status of the IP Management VLAN, meaning that if any port in the management VLAN is up, then the status of the management VLAN is reported as up. (Q02114651)
• The log events created on non-base units in a stack now correctly display the time when SNTP is enabled (Q02112485)
• VLACP settings are now correctly set when operating in a stack and the VLACP parameters are defaulted (Q02069051)
• When the member of a MLT group which was carrying multicast traffic is reset, IGMP membership is now correctly re-learnt without an unexpected delay of 40 second (Q02066898)
• When IGMP Snooping and Proxy is enabled in a stack if the IGMP general query and IGMP host membership report are received on the same unit in the stack, IGMP packets are now correctly processed (Q02070898)

Please review the release notes for all the details.

Cheers!

You can certainly do this from Nortel’s Java Device Manager, however, you need to be careful that you don’t saw off the branch you’re standing on when you change the SNMP community string. It’s best to configure the SNMP community strings from the CLI interface to avoid any potential issues.

Here are the CLI commands to configure the SNMP community strings on the ERS 8600 and 1600 switch. In the example below we’ll set the read-only string to open and the read-write string to lock.

ERS-8610:5# config snmp-v3 community commname first new-commname open
ERS-8610:5# config snmp-v3 community commname second new-commname lock

Here are the CLI commands to configure the SNMP community strings on the ERS 4500, ERS 5500 and ES460/470 switches. In the example below we’ll set the read-only string to open and the read-write string to lock.

5520-48T-PWR (config)# snmp-server community open ro
5520-48T-PWR (config)# snmp-server community lock rw

Cheers!

I should point out that you can’t just load 3.7.4 software if you are running software older than 3.5.4. You must upgrade the software in steps. Here’s the different versions that you need to step through; 2.x to 2.5 to 3.0 to 3.2.1 to 3.5.4 to 3.7.4.

Also of interest in the release notes is a comment regarding VLACP;

VLACP operation has been changed to match implementation on ERS modular and other stackable platforms with the latest software releases. A link for which VLACP is configured will remain in a blocked state until partnership is correctly formed with the VLACP partner (Q01799355).

I’m not sure what difference they are referring to… the wording makes it sound as though VLACP would initial start in forwarding mode and would only block the port once the timeout and retries had reached the configured threshold. I might need to dig a little deeper just out of curiosity.

You can find the release notes here.

After writing this article for the past 90 minutes I decided to break it into multiple parts for multiple reasons, firstly because it’s getting long and secondly I don’t want to skimp on the content and want to get it right.

Example 1 – Ethernet Routing Switch 8600 to Ethernet Switch 470 using LACP trunk

In this example we’ll build an LACP trunk between a Nortel Ethernet Routing Switch 8600 and a Nortel Ethernet Switch 470. As we all know a picture is worth a thousand words so let’s start with a simple basic diagram of our two switches;

We’ll start with the Ethernet Routing Switch 8600 and I’ll walk you through the commands.

In our first step we’ll create the MultiLink Trunk (MLT) table entry which is required for the ERS 8600, it’s not required for the the majority of Nortel’s other switches. We’ll be using LACP key 33, I chose 33 for no real specific reason. The value is important though because it also needs to be unique (not already used) and needs to be used later in the configuration. This value will identify the ports in the switch that should participate in the trunk configuration. You can have multiple LACP LAGs each with their own key, sometimes referred to as the admin key.

config mlt 13 create
config mlt 13 name "LACP-LAG"
config mlt 13 lacp key 33
config mlt 13 lacp enable

The next step is to configure the actual ports that will make up the trunk. In my example I’m using ports 7/33 and 7/34, again for no specific reason. I’ll enable tagging if for no other reason than to preserve any 802.1q headers such as the Priority Code Point (PCP) which is just the 802.1p bits. I’ll also add the ports to VLAN 99 which we’ll bridge between the two switches;

config ethernet 7/33-7/34 perform-tagging enable
config vlan 1 ports remove 7/33-7/34
config vlan 99 port add 7/33-7/34

With the basic port configuration complete we now need to turn our attention to the LACP specific parameters. We need to use the same key we used to create the MLT above.

config ethernet 7/33-7/34 lacp key 33
config ethernet 7/33-7/34 lacp timeout short
config ethernet 7/33-7/34 lacp aggregation true
config ethernet 7/33-7/34 lacp enable

Let’s not forget to enable LACP globally;

config lacp enable

That’s it.

Hopefully you’ll agree that it isn’t too hard. Now let’s focus on the configuration steps for the Ethernet Switch 470. First we’ll enable tagging on the ports we’re going to use on the Ethernet Switch 470;

vlan ports 33,34 tagging tagAll

Let’s add VLAN 99 to the ports, I’ve already created the VLAN ahead of time.

vlan members add 99 33,34

Now we just need to configure the LACP parameters for each port and then enable LACP.

interface fastEthernet 33-34
lacp key 13
lacp mode active
lacp timeout-time short
lacp aggregation enable
exit

That’s it.

Let’s just make sure that everything is working properly on both the 8600 and the 470 switches.

ERS-8610:6# show mlt info
================================================================================
Mlt Info
================================================================================
PORT    SVLAN  MLT   MLT        PORT         VLAN
MLTID IFINDEX NAME      TYPE    TYPE  ADMIN CURRENT    MEMBERS      IDS
--------------------------------------------------------------------------------
13  6156  LACP-LAG     trunk   normal norm   norm     7/33-7/34         99

MULTICAST             DESIGNATED   LACP      LACP
MLTID IFINDEX  DISTRIBUTION  NT-STG  PORTS        ADMIN     OPER
--------------------------------------------------------------------------------
13     6156     disable      disable  7/33       enable      up

ERS-8610:6# show mlt lacp info
================================================================================
LACP Aggregator Information
================================================================================
MAC               COLLECTOR     AGGR    PORT
MLTID IFINDEX       ADDR              MAXDELAY     ORINDI   MEMBERS
--------------------------------------------------------------------------------
13    6156       00:0f:cd:f1:e1:30    32768        aggr    7/33-7/34

--------------------------------------------------------------------------------
OPER        MIN        OPERLAST
MLTID IFINDEX     STATE       LINK       CHANGE
--------------------------------------------------------------------------------
13    6156        up          1          6 day(s), 09:54:53

--------------------------------------------------------------------------------
ACTOR      ACTOR          ACTOR      ACTOR
MLTID IFINDEX    SYSPRIO     SYSID         ADMINKEY   OPERKEY
--------------------------------------------------------------------------------
13    6156       32768   00:0f:cd:f1:e0:00    33      33

--------------------------------------------------------------------------------
PARTNER      PARTNER        PARTNER
MLTID IFINDEX    SYSPRIO      SYSID          OPERKEY
--------------------------------------------------------------------------------
13    6156       32768   00:16:60:80:07:c0    8205

And now on the 470 switch;

ES-470#show mlt
Trunk Name                 Members             Bpdu   Mode   Status
----- -------------------- ------------------- ------ ------ --------
1     Trunk #1             NONE                All    basic  Disabled
2     Trunk #2             NONE                All    basic  Disabled
3     Trunk #3             NONE                All    basic  Disabled
4     Trunk #4             NONE                All    basic  Disabled
5     Trunk #5             NONE                All    basic  Disabled
6     Trunk #6             33-34               Single DynLag Enabled

ES-470#show lacp port 33,34
Admin Oper         Trunk Partner
Port Priority Lacp    A/I Timeout Key   Key   AggrId Id    Port    Status
---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
33   32768    Active  A   Short   13    8205  8193   6     480     Active
34   32768    Active  A   Short   13    8205  8193   6     481     Active

I think that’s enough for now… there’s a simple LACP configuration between two switches. Please know that you cannot add/remove VLANs from LACP LAG configurations dynamically on Nortel switches. You must disable the LACP configuration and then change the port configuration. So there’s a great benefit to use Nortel’s proprietary MultiLink Trunk (MLT) where possible. Please also note that you don’t need to enable VLACP since LACP already performs the same functionality provided by VLACP.

Please feel free to post specific questions in the discussion forums;
http://forums.networkinfrastructure.info/nortel-ethernet-switching/

Cheers!

HP’s NIC Teaming now supports 802.3ad (LACP) which provides a load balanced and fault tolerant solution. I’ve posted several documents to my website for anyone that might be looking for additional information.

Here is a very good document that outlines NIC teaming from head to toe, along with additional documents that outline the different capabilities and what operating systems are supported.

In short you need to setup an LACP group on the Nortel switch for any ports that you want to be in the LAG (Link Aggregation Group). You can leave the HP NIC teaming options set to Automatic and it should negotiate as a “802.3ad Dynamic with Fault Tolerance” when you have the Nortel switch configuration properly.

5520-48T-PWR> enable
5520-48T-PWR# config term
5520-48T-PWR(config)# interface FastEthernet 1/1-2
5520-48T-PWR(config-if)#lacp aggregation port 1/1-2 enable
5520-48T-PWR(config-if)#lacp mode port 1/1-2 active
5520-48T-PWR(config-if)#lacp key port 1/1-2 101

Cheers!

ERS5520-PWR#reload ?
cancel           Cancel a previous scheduled reload
force            Do not ask for confirmation
minutes-to-wait  Minutes to wait before reboot
<cr>

Cheers!

The bulletin advises users to re-configure the VLACP timeout from a default value of 3 to 5.

5520-48T-PWR(config)#interface fastEthernet 5-6
5520-48T-PWR(config-if)#vlacp timeout-scale 5
5520-48T-PWR(config-if)#show vlacp interface 5-6
===============================================================================
VLACP Information
===============================================================================
PORT ADMIN OPER HAVE FAST SLOW TIMEOUT TIMEOUT ETH MAC
ENABLED ENABLED PARTNER TIME TIME TYPE SCALE TYPE ADDRESS
-------------------------------------------------------------------------------
5 true true yes 500 30000 short 5 8103 00:00:00:00:00:00
6 true true yes 500 30000 short 5 8103 00:00:00:00:00:00

The bulletin also refers to a software fix in ERS 2500 v4.2.1, ERS4500 v5.2.1 and ERS5500/5600 v6.0.2 or later maintenance releases.

We really only use VLACP as a means of detecting FEFI when the switch equipment doesn’t support autonegotiation (example; Nortel Ethernet Switch 470 doesn’t support autonegotiation on the 1000Mbps uplinks).

Cheers!

Update: Friday February 13, 2009

It seems that Nortel has released software 6.0.3 for the Ethernet Routing Switch 5500/5600 series switches. This release is suppose to resolve the VLACP issues that were reported in the earlier bulletin. Here’s an excerpt from the release notes;

A feature enhancement (Q01645430) that changed the VLACP interoperability behavior with Passport 8600 was removed. For further details, please see the Technical Support Bulletin ID. 2008009238, Rev 1, published on 2008-12-12.

Cheers!

I’ve used the Nortel Discovery Protocol on a number of occasions to help document and troubleshoot problems within a network. While Nortel’s Java Device Manager (GUI) provides support for displaying the topology table it leaves some very vital information out, specifically the remote card and port from where the connection is originating. You can however, view that information from the CLI interface of Nortel’s Ethernet Switches (ES) and Ethernet Routing Switches (ERS).

Here’s an example of the topology table from an Ethernet Routing Switch 5530 stack which is Split MultiLink Trunk (SMLT) connected to a cluster pair of Ethernet Routing Switch 8600s;

5530-24TFD#show autotopology nmm-table
LSlot                                                                     RSlot
LPort IP Addr          Seg ID  MAC Addr     Chassis Type     BT LS   CS   RPort
----- --------------- -------- ------------ ---------------- -- --- ----  -----
0/ 0 10.102.255.65   0x000000 00159BEACC00 5530-24TFD       12 Yes HTBT    NA
1/23 10.102.1.5      0x000406 0004387070E8 Passport 8610    12 Yes HTBT   4/ 6
2/47 10.102.1.6      0x000406 000FCDF1E0E8 Passport 8610    12 Yes HTBT   4/ 6

You can see from the information above that ports 1/23 and 2/47 on the ERS 5530 connect to port 4/6 on the ERS 8600 Core A (10.102.1.5) and port 4/6 on the ERS 8600 Core B (10.102.1.6).

Looking at one of the core ERS 8600 switches we can see the following topology table;

ERS8600:5# show sys topology

================================================================================
Topology Table
================================================================================
Local                                                                     Rem
Port  IpAddress       SegmentId MacAddress   ChassisType      BT LS  CS   Port
--------------------------------------------------------------------------------
0/0  10.102.1.5      0x000000  000438707000 ERS8610          12 Yes HtBt  0/0
1/1  10.102.1.6      0x000101  000fcdf1e000 ERS8610          12 Yes HtBt  1/1
1/5  10.102.255.19   0x00012f  001e7e7b0c01 mBayStack4500-48GT-PWR 12 Yes HtBt  1/47
1/6  10.102.255.35   0x000130  000cf73c25c1 mBayStack470     12 Yes HtBt  1/48
1/7  10.102.255.60   0x00012f  0014c733e401 mBayStack5520-48T-PWR 12 Yes HtBt  1/47
2/20 10.102.1.9      0x000201  001d427b7040 ERS8610          12 Yes HtBt  2/1
4/1  10.102.1.6      0x000401  000fcdf1e0c0 ERS8610          12 Yes HtBt  4/1
4/4  10.102.255.45   0x000119  0011f9abc541 mBayStack470-24T 12 Yes HtBt  1/25
4/6  10.102.255.65   0x000117  00159beacc00 mERS5530-24TFD   12 Yes HtBt  1/23
4/7  10.102.255.75   0x000132  000e40eb4031 Passport1648     12 Yes HtBt  1/50
9/1  10.102.255.25   0x000119  00802deb6150 mBayStack450     12 Yes HtBt  1/25

You can see from this table that there are quite a few edge/closet switches connected to this specific ERS 8600 and you can quickly and easily identify which ports they are connected to.

Cheers!

Cheers!

There are of course some caveats, aren’t there always caveats!. You can only stack specific switches with themselves. For instance you can only stack a Nortel Ethernet Routing Switch 5500 series switch (5510, 5520, 5530) with anther Nortel Ethernet Routing Switch 5500 series. You can’t stack a Nortel Ethernet Routing Switch 4500 series with a Nortel Ethernet Routing Switch 5500 series. The stack cables are very different so it shouldn’t be too hard to figure out. The switches must have the same software version of code on them in order to stack properly. The newer Ethernet Routing Switch 4500/5500 series switches will try to automatically upgrade any switch that is added to the stack and isn’t running the appropriate software version.

You can cascade/stack older Ethernet Switch 450, Ethernet Switch 460 and Ethernet Switch 470 switches. I would not advise stacking the Ethernet Switch 450 with either the ES460 or ES470 even though Nortel claims you can. There’s no issue stacking a Nortel Ethernet Switch 460 with the Ethernet Switch 470.

Cheers!

Let me describe a typical scenario and then offer some ways of isolating the potential problem. You have a stack of four ES470s we’ll refer to as Unit 1, Unit 2, Unit 3 and Unit 4. We can use the picture to the left to visualize what a stack of four Ethernet Switch 470s might look like. While all the Up/Down stack lights should be green let’s just say that Unit 3 Down and Unit 4 Up is amber.

Let me just warn you that I have yet to figure how to truly identify a bad cascade module (the module that is built into the switch) from a bad cable without using either a cascade module that is know to be good and/or cascade cable that is know to be good in a process of elimination.

How you can determine if you have a bad cascade cable or cascade module?

It’s really pretty easy although it will require you to take the switch down and use the diagnostic boot code. You’ll need to cable up to the serial interface of the switch in order to run the test. When you’re ready go ahead and cold boot the switch. When you see the following, “470-24T  Diagnostics 3.6.0.7” (or something similar since you may not have a 24T but a 48T) you’ll need to interrupt the boot sequence by hitting Ctrl-C (go ahead and hit it repeatedly). You should see something similar to the following;

470-24T  Diagnostics 3.6.0.7

Testing main memory - PASSED

>> Break Recognized - Wait..

>> Break Recognized - Wait..

Press 'a'  to run Agent code
Press 'c'  to run Cascade external loopback test
Press 'd'  to Download agent code
Press 'e'  to display Errors
Press 'i'  to Initialize config/log flash
Press 'p'  to run POST tests
Press 'r'  to Receive cascade test packets
Press 's'  to Send    cascade test packets..

Once your at this point you’ll need to take a single cascade cable and loop it between the Up and Down port of the switch your working on. This will put a physical loop between the two interfaces so we can run and external loopback test across the cascade links. When you’re ready go ahead and select “c” from the diagnostics menu.

Test 501  Stack External Loopback    -        FAILED
NSX SXLB STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7

In my case the Ethernet Switch 470 24 Port switch that I was using failed the loopback test. I then took a cascade cable that I knew to be working and repeated the test. It subsequently failed again which indicates to me that the cascade module is faulty. If you were to select “e” from the diagnostics menu you might seem something similar to the following;

System Resets  =       58.

Burn-In Loops  =        0.
Burn-In Errors =        0.
Auto-Burn-In   = DISABLED
Diag Baud      =     9600.

Error Log:
Bad Port Mask  = 80000000
Loop Test Error Description:
  50  501 STAK: Stack Secondary Rx (1) Timed Out
  50  501 STAK: Stack Upstream Clock Failed. Is Cascade Cable Missing?
  50  501 STAK: Stack Secondary Rx (1) Timed Out
  50  501 STAK: Stack Secondary Rx (1) Timed Out
  50  501 STAK: Force Stack RNGO Low Failed Test=0 GCReg=60
  50  501 STAK: Force Stack RNGO Low Failed Test=0 GCReg=60
  56  501 STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7
  56  501 STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7
  58  501 STAK: Stack Upstream Clock Failed. UCR=27 DCR=A7

One very important note! You can only stack switches that are running the same version of software (boot code and agent code). I believe the “Base” light will blink amber if you try to stack two switches together that are not running the same software.

You can also confirm a cascade/stacking issue remotely using Nortel’s Device Manager. Here’s a screenshot of two Ethernet Switch 470s stack together. You can see the yellow LEDs on Unit 1 Up and Unit 2 Down.

I will let you know that we’ve had our own share of cascade modules go bad over the past five years. While the cascade modules appear to be “replaceable” they really not designed to be field serviceable. If a switch fails the cascade loopback test it’s really only good for stand alone operation.

Cheers!

I recently received a message from someone looking for someway to automated the re-configuration of over 100 switches with the correct Daylight Saving Time configuration. I explained to the person that the best long term solution would probably be to use the SNMP MIB but a quick and dirty solution might be to use Expect and call it from a Bash script looping over all the switches that needed to be re-configured. In short Expect is a scripting language that mimics user input at a TTY.  The Except script is written to issue a set of commands, as if a human were typing them, and expects various responses.

The script I wrote below only support a limited number of switches. If you have a particular switch you’re welcome to modify the script to support that particular switch. The script will attempt to determine if the switch is running the software that has the features we’re looking to implement. I didn’t have a whole lot of time to test so buyer beware!

Here’s the expect script that I authored;

#!/usr/bin/expect -f
#
##############################################################################
#
# Filename: /usr/local/etc/set-nortel-timezone.exp
#
# Purpose:  Expect script designed to telnet into Nortel Ethernet Switches
#           and execute the CLI commands to confgure the appropriate timezone
#           information, including Day Light Saving time.
#
# Switches: Ethernet Switch 460 v3.7.x
#           Ethernet Switch 470 v3.7.x
#           Ethernet Switch 4500 v5.2.x
#           Ethernet Switch 5500 v5.1.x
#
# Author:   Michael McNamara
#
# Date:     June 1, 2008
#
# Version:  1.1
#
# Changes:
#
#           June 8, 2008 (M.McNamara)
#           - added documentation and ARGV command line checks
#           June 14, 2008 (M.McNamara)
#           - added check for switch version and exit if v3.6 switch software
#           - added check for Username introduced in v3.7 switch software
#
#
##############################################################################
#
# This Expect script was generated by autoexpect on Wed Jul 27 17:25:28 2005
# Expect and autoexpect were both written by Don Libes, NIST.
#
set force_conservative 1  ;# set to 1 to force conservative mode even if
                          ;# script wasn't run conservatively originally
if {$force_conservative} {
        set send_slow {1 .1}
        proc send {ignore arg} {
                sleep .1
                exp_send -s -- $arg
        }
}

if {[llength $argv] != 2} {

   puts "usage: set-nortel-timezone.exp < SWITCH > < PASSWORD >>"

exit 1

}

#
set PATH "/usr/local/etc/"
set TELNET "/usr/bin/telnet"

set SWITCH [lindex $argv 0]
set PASSWORD [lindex $argv 1]

set TODAY [timestamp -format %y%m%d ]
set WEEKDAY [timestamp -format %a ]
set DATE [timestamp -format %c ]

set send_human {.1 .3 1 .05 2}

#log_file $PATH/$SWITCH.expect.log
log_file /usr/local/etc/password.expect.log
log_user 0      # Disable logging to STDOUT
#log_user 1     # Enable logging to STDOUT

set timeout 10
spawn $TELNET $SWITCH
match_max 100000

expect "Trying"
expect {
   "Connected"  {

      expect "SW:v3.6" {
         send_log "\n\nThis version of software doesn't support the CLI commands!\n"
         send_user "\n\nThis version of software doesn't support the CLI commands!\n"
         exit 1
      }
      sleep 1
      send -- ""
                }
   Timeout      {
      send_log "We're unable to connect to the switch $SWITCH"
      send_user "We're unable to connect to the switch $SWITCH"
      exit 1;
                }
}

expect {
   "Username"   {
      send -- "RW\r"
   }
}

expect "Enter Password"
send -- "$PASSWORD\r"

expect {
   "Main Menu"  {
                }
   "Incorrect Password" {
      send_log "$SWITCH : Incorrect Password"
      exit 1
   }
   "Incorrect Credentials" {
      send_log "$SWITCH: Incorrect Credentials"
      exit 1
   }
}
sleep 1

# Let's get into the CLI interface from the menu prompts
send -- "C"

# Depending on the version of software we sometimes need a CR/LF
send -- "\r"
sleep 1

# Let's wait for the CLI prompt which includes the #
expect "#"
send -- "config term\r"
send -- "clock time-zone EST -5\r"
send -- "clock summer-time EDT date 9 Mar 2008 2:00 2 Nov 2008 2:00 +60\r"
send -- "exit\r"
send -- "logout\r"
expect eof

You can download the entire Expect script from this URL; set-nortel-timezone.exp.

The command line arguments are fairly straight forward;

usage: set-nortel-timezone.exp <SWITCH> <PASSWORD>

Where the SWITCH is the fully qualified domain name (FQDN) or the IP address of the switch in question and the PASSWORD is the Read-Write password for the switch.

If you had hundreds of switches to reconfigure you could wrap this Except script in a Bash shell script similar to the following;

#!/bin/bash
#
#####################################################################
#
# Language: Bash Shell Script
#
# Filename: /usr/local/etc/set-nortel-timezone.sh
#
# Purpose:  This script will kickoff the Expect script that will
#           configure the Daylight Saving Time features for each switch
#
# Author:   Michael McNamara
#
# Date:     June 1, 2008
#
# Version:  1.0
#
# Changes:
#
#           June 10, 2006 (M.McNamara)
#           -  added remote sites into shell script processing
#
#####################################################################
#

# Variables
PATH_TO=/usr/local/etc
UPGRADE=set-nortel-timezone.exp
MAIL_LIST=''
PAGER_LIST=''
ERROR_FLAG=0
MAILEXE='/usr/bin/mutt'
LOCKFILE=/tmp/trace.lck

# Check paramaters
if [ "$#" != 2 ]
then
  echo "Usage: `basename $0` <password>"
  exit 1
fi

PASSWORD=$1

#####################################################################
#####################################################################
# YOU SHOULD EDIT THE "SWITCHES" VARIABLE BELOW TO INCLUDE ALL THE
# SWITCHES THAT YOU WISH TO HAVE THE EXPECT SCRIPT RUN AGAINST
#####################################################################
#####################################################################

SWITCHES='sw1-5520.acme.org sw2-5520.acme.org sw3-5520.acme.org'

for SWITCH in $SWITCHES
do
        $PATH_TO/$UPGRADE $SWITCH $PASSWORD
done

exit

You can download the Bash shell script from this URL; set-nortel-timezone.sh.

I’ve only tested this on CentOS v5.2 but it should work on any Linux host with Expect installed although you may need to modify the path locations.

Cheers!

We generally perform software upgrades on all our routers and switches twice a year. It really helps to keep our network infrastructure current and it also helps to reduced unscheduled downtime.

Last fall we decided to skip the bi-yearly maintenance because there were just too many projects on the docket. This spring we came across a very interesting issue that we had never seen in the past. We started to notice that multiple Nortel Ethernet Switch 460/470 switches/stacks were rebooting themselves all over our network. It took us a few hours to realize that every switch that had rebooted had just eclipsed approximately 500 days of uptime. All the affected switches were running FW 3.6.0.6 with SW v3.6.4.08. The switches were literally rebooting themselves in the same order in which they had been upgraded almost 500 days earlier.

I’m currently trying to confirm with Nortel that this “bug” has been removed from the 3.7.x software release.

This was one occasion where the network was just too good for itself.

Cheers!

Update: Tuesday June 10, 2008

I received a formal response from Nortel today that included the following:

Analysis of the issue :-
When the BS-470 switches reaches 497 days the system time rolls over and during this period management communication will be lost. This is caused by the use of a 32 bit counter, which when it rolls back to 0, initiates an internal software synchronization to align all timers. This is only loss of IP management and not switching functionality.

This issue still open and can be fixed by rebooting the switches before reaching the 497 day mark.

When I inquired if the problem had been resolved in the v3.7.x software release I was told it had not. It would seem that a lot of folks just don’t expect switches to be running that long these days.

Cheers!

Update: Wednesday November 4, 2008

Last week Nortel released a technical service bulletin entitled, “Ethernet Routing Switches: SysUpTime approaching 497 days can cause the switch or stack to behave in some unexpected way“. They also released a video that documents a workaround to the problem.

Let me save you the time and effort of downloading either. Nortel solution is truely masterful; reboot the switch.

While I’ve been know to defend Nortel there’s just no defense for this. I’m completely floored at Nortel’s response.

Cheers!

Note: I’m still trying to figure out the best way to display the CLI stuff… if I use the PRE HTML tag the font is really too small, if I don’t use the PRE HTML tag the formatting (spacing) gets lost making it difficult to compare the post with the real world output from a CLI interface.

Nortel Ethernet Routing Switch 5500 Series (v5.1)

Here’s how to set the passwords on the Nortel Ethernet Routing Switch 5500 Series (v5.1 software).

5520-48T-PWR>enable
5520-48T-PWR#config term
Enter configuration commands, one per line.  End with CNTL/Z.

What’s the syntax to set the read-only and read-write passwords?

5520-48T-PWR(config)#cli password ?
read-only   Modify read-only password
read-write  Modify read-write password
serial      Enable/disable serial port password.
telnet      Enable/disable telnet and web password.

We’ll use the commands below to set the read-only (RO) password to “readonlypassword” and the ready-write (RW) passwords to “readwritepassword”;

5520-48T-PWR(config)#cli password read-only readonlypassword
5520-48T-PWR(config)#cli password read-write readwritepassword

What is the syntax to enable the passwords on the serial and telnet interfaces?

5520-48T-PWR(config)#cli password serial ?
local   Use local password.
none    Disable password.
radius  Use RADIUS password authentication.
tacacs  Use TACACS+ AAA services

5520-48T-PWR(config)#cli password telnet ?
local   Use local password.
none    Disable password.
radius  Use RADIUS password authentication.
tacacs  Use TACACS+ AAA services

We’ll use the commands below to set the serial and telnet interface to use the local passwords we’ve just configured above. You could also use RADIUS and TACACS authentication if you set it up.

5520-48T-PWR(config)#cli password serial local
5520-48T-PWR(config)#cli password telnet local

And let’s not forget to save the configuration file (even though the switch should auto-save it).

5520-48T-PWR(config)#copy config nvram
5520-48T-PWR(config)#exit
5520-48T-PWR#disable
5520-48T-PWR>

Nortel Ethernet Routing Switch 4500 Series (v5.0)

The Nortel Ethernet Routing Switch 4500 Series (v5.0 software) is piratically identical to the 5500 series except that it does not yet support TACACS authentication.

4548GT-PWR(config)#cli password ?
read-only   Modify read-only password
read-write  Modify read-write password
serial      Enable/disable serial port password.
telnet      Enable/disable telnet and web password.

4548GT-PWR(config)#cli password serial ?
local   Use local password.
none    Disable password.
radius  Use RADIUS password authentication.

4548GT-PWR(config)#cli password telnet ?
local   Use local password.
none    Disable password.
radius  Use RADIUS password authentication.

Nortel Ethernet Switch 460/470 (v3.7.2)

The Nortel Ethernet Switch 460/470 (v3.7.2 software) is identical to the ERS 4500 series.

470-48T>enable
470-48T#config term
Enter configuration commands, one per line.  End with CNTL/Z.

470-48T(config)#cli password ?
read-only   Modify read-only password
read-write  Modify read-write password
serial      Enable/disable serial port password.
telnet      Enable/disable telnet and web password.

470-48T(config)#cli password serial ?
local   Use local password.
none    Disable password.
radius  Use RADIUS password authentication.

470-48T(config)#cli password telnet ?
local   Use local password.
none    Disable password.
radius  Use RADIUS password authentication.

Hopefully this should help a few folks out.

Cheers!

• Nortel Business Policy Switch 2000
• Nortel Ethernet Switch 300 Series
• Nortel Ethernet Switch 460
• Nortel Ethernet Switch 470
• Nortel Ethernet Switch 2500 Series
• Nortel Ethernet Switch 4500 Series
• Nortel Ethernet Route Switch 5500 Series

It is possible that the switch agent image, that is stored in NVRAM on the switch, can become corrupt for some reason or another. In this case the switch will not boot up properly and will require some special intervention.

Diagnostic Version X.X.X.X
Press Control-C to Enter Diag

Test ROM Config - PASSED
Test FANs - PASSED
Test Internal Loopback - PASSED
Test ASIC1 Registers - PASSED
Test ASIC2 Registers - PASSED
Test PHY Registers - PASSED
Test USB Registers - PASSED
Agent code verification fails!

>> Break Recognized - Wait...
Press 'a' to run Agent code
Press 'd' to download Agent code
Press 'e' to display errors
Press 'c' to clear log message
Press 'i' to initialize config flash
Press 'p' to run POST tests...

You can use the boot diagnostic code to download the agent code to the switch using the “d” option. While the switch is booting using “Ctrl-C” to break the boot sequence and select “d” from the menu.

WARNING: the TFTP server needs to be physically connected to the switch in question

Download Agent Code

Enter Port Number [  ]:
Enter Speed: 10, 100, 1000 [  ]:
Enter Local IP Address [ 0.0.0.0 ]: 10.10.10.15 (IP given to Switch)
Enter Server IP Address [ 0.0.0.0 ]: 10.10.10.1 (IP of local TFTP server)
Enter Subnet Mask [ 255.255.255.0 ]:
Enter Filename: boss_1234.img
Wait..
TFTP: Sending Open: .aaaaa.a
TFTP: Open
...............................................................
.............................................................
Len= 0x20795E= 2128222. (@1200000)
Agent Version= 5.0.0.0 ModelMask= 0x1C
Program y/N [ N ]: y (Press y)
Erasing - Wait 56 sec..
Programming - Wait 96 sec..

Once the download is complete you’ll need to run the agent code by selecting “a”

Starting Agent Code..

Decompressing the image ...
Target Name: vxTarget
User: target
Attaching network interface idtip0... done.
Attaching network interface lo0... done.

Completing initialization...

At this point the switch should be booting up although it may take ~ 2 minutes for the switch to fully initialize the software and configuration.

Cheers!

Ethernet Switching Feature Matrix July 2007 Public Version.pdf

This is public information so hopefully I won’t be getting any nasty email messages from anyone.

Cheers!

UPDATE: April 3, 2008

Here’s a new version of the Ethernet Switching Feature Matrix dated November 2007.

In software release v3.7.x for the Nortel Ethernet Switch 460/470 you’ll also find that you now need to provide a username when you telnet into the switch (in previous releases you were only prompted for a password, now you are prompted for a username and password).

Interestingly enough you cannot change the default usernames (at least I don’t believe you can).

For the above mentioned switches there are only two levels of access, read-write and read-only.

The default username for the read-write user level is RW.
The default username for the read-only user level is RO.

Updated 1/16/08: I should have included the default passwords for those two accounts.
The default password for the read-write user level is “secure”.
The default password for the read-only user level is “user”.

Cheers!

As with the previous procedure you’ll need access to the console port on the switch. Specifically you’ll need to cable up (9600,8,N,1) to the CPU (8690SF, 8691SF, 8692SF) you wish to reset.

If you’ve lost the password… cold boot the chassis while connected to the console port. When the switch starts to boot you should see something similar to the following (depending on the version of software installed);

Copyright (c) 2007 Nortel, Inc.
CPU Slot 5:    PPC 745 Map B
Version:       4.1.5.4
Creation Time: Dec 17 2007, 15:31:21
Hardware Time: DEC 26 2007, 16:19:24 UTC
Memory Size:   0x10000000
Start Type:    cold
SMI ZOOMCF
can't open "/pcmcia/pcmboot.cfg" 0x380003
S_dosFsLib_FILE_NOT_FOUND
/flash/  - Volume is OK
Change volume Id from 0x0 to 0x1a5

Loaded boot configuration from file /flash/boot.cfg
Attaching network interface lo0... done.

Press  to stop auto-boot...
1

You’ll need to interrupt the boot process by hitting the “Return” key . You should be greeted with a monitor prompt;

monitor#

From here you’ll be able to issue a command to clear the passwords stored in NV RAM;

monitor# reset-passwd
monitor#

Now just go ahead and reset the CPU and you should be able to login with the default username (rwa) and password (rwa).

monitor# reset

CPU Slot 5:    PPC 745 Map B
Version:       4.1.5.4
Creation Time: Dec 17 2007, 15:31:21
Hardware Time: DEC 26 2007, 16:25:09 UTC
Memory Size:   0x10000000
Start Type:    cold
SMI ZOOMCF
can't open "/pcmcia/pcmboot.cfg" 0x380003
S_dosFsLib_FILE_NOT_FOUND
/flash/  - Volume is OK
Change volume Id from 0x0 to 0x1a5

Loaded boot configuration from file /flash/boot.cfg
Attaching network interface lo0... done.

Press  to stop auto-boot...
Loading /flash/p80a4154.img ... 8761414 to 25459172 (25459172)
Starting at 0x10000...

SMI ZOOMCF
Booting PMC280 Mezz HW please wait
. The BootCode address is 0x2b00100 3303
.
Mezz taking over console and modem......
Mezz CPU Booted successfully

Initializing backplane net with anchor at 0x4100... done.
Backplane anchor at 0x4100... ..
Mounting /flash: .done.

Ethernet Routing Switch 8600  System Software Release 4.1.5.4
Copyright (c) 1996-2007 Nortel, Inc.

CPU5 [10/26/99 11:26:25] SW INFO System boot
CPU5 [10/26/99 11:26:25] SW INFO ERS System Software Release 4.1.5.4
CPU5 [10/26/99 11:26:26] SW INFO CPU card entering warm-standby mode...
CPU5 [10/26/99 11:26:27] SW INFO Loading configuration from /flash/config.cfg

CPU5 [10/26/99 11:26:27] SW INFO PCMCIA card detected in Stand-by CPU "ERS-8610"
slot 5, Chassis S/N SSPND*****

**************************************************
* Copyright (c) 2007 Nortel, Inc.                *
* All Rights Reserved                            *
* Ethernet Routing Switch 8010                   *
* Software Release 4.1.5.4                       *
**************************************************

Login:

You should now be able to login with the default RWA username of “rwa” and the default password for “rwa”.

If you wish to reset the configuration… you only need to delete the config.cfg file from the flash and reset the switch.

You should NOT delete the boot.cfg file unless you have a copy of the software on the PCMCIA card and know how to start the software using the boot command from monitor mode.

I believe the same monitor command is available for the Ethernet Routing Switch 1600 Series.

Cheers!

Here’s what Nortel has to stay in their document, “Link Aggregation Control Protocol (LACP) 802.3ad and VLACP Technical Configuration Guide” dated August 2007;

Virtual LACP (VLACP) is an extension to LACP, used to detect end-to-end failure. VLACP takes the point-to-point hello mechanism of LACP and uses it to periodically send hello packets to ensure end-to-end reachability and provide failure detection (across any L2 domain). When Hello packets are not received, VLACP transitions to a failure state and the port will be brought down. The benefit of this over LACP is that VLACP timers can be reduced to 400 milliseconds between
a pair of ERS8600 switches. This will allow for approximately one second failure detection and switchover. Note that the lowest VLACP timer on an ES460/470 is 500ms. VLACP can also be used with Nortel’s proprietary aggregation mechanism (MLT) to complement its capabilities and provide quick failure detection. VLACP is recommended for all SMLT access links when the links are configured as MLT to ensure both end devices are able to communicate. By using VLACP over Single-Port SMLT, enhanced failure detection is extended beyond the limits of the number of SMLT or LACP instances that can be created on the ERS8600. VLACP can also be used as a loop prevention mechanism in SMLT configurations and should be used when setting up the IST. It also protects against CPU failures by causing traffic to be switched or rerouted to the SMLT peer in the case the CPU fails or gets hung up. Please refer to the Technical Configuration Guide for Switch Clustering using Split-Multilink Trunking (SMLT) with ERS8600 for more details.

NOTE: In regards to the ERS8600, although either the CLI or JDM interface allows you to configure the short timers to less than 400ms, Nortel does not support this configuration unless the ERS8600 is equipped with the SuperMezz daughter module for the 8692SF. The SuperMezz allow for very quick sub 100ms failure detection.

Although functions such as Remote fault indication (RFI) or Far-end fault indication (FEFI) can be used to indicate link failure, there are some limitations with these mechanisms. The first limitation is that with either of these mechanisms, they terminate at the next Ethernet hop. Hence, failures cannot be detected on an end-to-end basis over multiple hops such as LAN Extension services. The second limitation is both of these mechanisms required Auto-Negotiation to be enabled on the Ethernet interface. Hence, if an Ethernet interface does not support Auto-Negotiation; neither of these mechanisms can be used. The third limitation is if an Ethernet interface should fail and still provide a transmit signal, RFI nor FEFI will be able to detect a failure. Hence, the far-end interface will still think the link up and continue to transmit traffic. VLACP will only work for port-to-port applications when there is a guarantee for a logical port-port match. It will not work in a port-to-multi-port scenario where there is no guarantee for a pointpoint match.

NOTE: Please note that VLACP does not perform link aggregation. Is it simply used to detect end-to-end link failures and can be enabled over single links or even MLT trunks. VLACP does not require LACP to be enabled; LACP and VLACP are independent features.

NOTE: When configuring VLACP, both ends of the link must be configured with the same EtherType, Multicast MAC address, and same timers. By default, the VLACP parameters across all ES and ERS switches are the same with the exception of the FastPeriodicTimer which is set to 200ms on the ERS8600 and 500ms on all other switches. When connecting, for example, an ERS8600 to and ERS5500, the recommendation is to use 500ms FastPeriodicTimers with ShortTimeout in order to achieve fast failover. Also, when using the ES460/470 in the 3.6.x software release, the VLACP EtherType must be configured with a different value on each MLT link. The EtherType must match the EtherType value at the far end of the MLT link.

NOTE: If VLACP is used with LACP, there is no difference in how VLACP and LACP bring down a port if no LACP or VLACP PDUs are received. VLACP will declare the VLACP status as down and will report the event in the log file whereas LACP will not synchronize, not activate Collecting and Distributing on this port, and not report a message in the log file. The end result is the same where the port will block traffic; the physical layer for this port will remain up. Although you can enable VLACP with LACP, there is no practical reason why you would do so.

There was an interim solution before VLACP developed by Nortel called Single Fiber Fault Detection (SFFD) specifically designed to allow remote fault detection on Gigabit Ethernet fiber ports that did not support autonegotiation. Unfortunately we had some issues with SFFD and never really deployed the feature beyond our testlab environment.

Ethernet Routing Switch 5510
Here’s how you would configure VLACP on the MLT uplinks to an ERS 8600 Switch. You’ll need to connect to the 5510 switch and enter the “Command Line Interface” if you have the menu up.

5510> enable
5510# configure terminal
5510(config)# interface fastEthernet 47,48
5510(config-if)# vlacp port 47,48 timeout short
5510(config-if)# vlacp port 47,48 enable
5510(config-if)# exit
5510(config)# vlacp enable
5510(config)# exit

Ethernet Routing Switch 8600
Here’s how you would configure VLACP on the MLT uplinks to the ERS 5510 Switch above.

ERS-8610:6# config ethernet 1/1, 2/1 vlacp enable
ERS-8610:6# config ethernet 1/1, 2/1 vlacp timeout short
ERS-8610:6# config ethernet 1/1, 2/1 vlacp fast-periodic-time 500
ERS-8610:6# config vlacp enable

In this example we’re using ports 1/1 and 2/1 as the uplinks to ports 47 and 48 on the ERS 5510 respectively. The VLACP short timeout timers on the ERS 8600 default to 200ms so we need to configure them to match the minimum possible with the ERS 5500 series switches of 500ms.

If the interface appears to be bouncing you should definitely check the timers.

Cheers!

Follow these steps:

1. Connect to the console port of the switch (9600,8,N,1)
2. Reboot the switch.
3. When the first line of the diagnostics tests is displayed, press CTRL-C. The system then displays a menu.
4. Select option “i” to factory default the switch.
5. Select option “a” to run the agent code.

Upon boot up, the switch will be in a factory default configuration.

Cheers!