Posts tagged ES460

Expect Script – Daylight Saving Time

5

O\'Reilly Exploring Expect

In one of my previous posts entitled, Network Time Protocol (NTP), I discussed how to setup a network time protocol sever and how to configure the Nortel Ethernet Switch and Ethernet Routing Switches for NTP including Daylight Saving Time (DST) support.

I recently received a message from someone looking for someway to automated the re-configuration of over 100 switches with the correct Daylight Saving Time configuration. I explained to the person that the best long term solution would probably be to use the SNMP MIB but a quick and dirty solution might be to use Expect and call it from a Bash script looping over all the switches that needed to be re-configured. In short Expect is a scripting language that mimics user input at a TTY.  The Except script is written to issue a set of commands, as if a human were typing them, and expects various responses.

The script I wrote below only support a limited number of switches. If you have a particular switch you’re welcome to modify the script to support that particular switch. The script will attempt to determine if the switch is running the software that has the features we’re looking to implement. I didn’t have a whole lot of time to test so buyer beware!

Here’s the expect script that I authored;

#!/usr/bin/expect -f
#
##############################################################################
#
# Filename: /usr/local/etc/set-nortel-timezone.exp
#
# Purpose:  Expect script designed to telnet into Nortel Ethernet Switches
#           and execute the CLI commands to confgure the appropriate timezone
#           information, including Day Light Saving time.
#
# Switches: Ethernet Switch 460 v3.7.x
#           Ethernet Switch 470 v3.7.x
#           Ethernet Switch 4500 v5.2.x
#           Ethernet Switch 5500 v5.1.x
#
# Author:   Michael McNamara
#
# Date:     June 1, 2008
#
# Version:  1.1
#
# Changes:
#
#           June 8, 2008 (M.McNamara)
#           - added documentation and ARGV command line checks
#           June 14, 2008 (M.McNamara)
#           - added check for switch version and exit if v3.6 switch software
#           - added check for Username introduced in v3.7 switch software
#
#
##############################################################################
#
# This Expect script was generated by autoexpect on Wed Jul 27 17:25:28 2005
# Expect and autoexpect were both written by Don Libes, NIST.
#
set force_conservative 1  ;# set to 1 to force conservative mode even if
                          ;# script wasn't run conservatively originally
if {$force_conservative} {
        set send_slow {1 .1}
        proc send {ignore arg} {
                sleep .1
                exp_send -s -- $arg
        }
}

if {[llength $argv] != 2} {

   puts "usage: set-nortel-timezone.exp < SWITCH > < PASSWORD >>"

exit 1

}

#
set PATH "/usr/local/etc/"
set TELNET "/usr/bin/telnet"

set SWITCH [lindex $argv 0]
set PASSWORD [lindex $argv 1]

set TODAY [timestamp -format %y%m%d ]
set WEEKDAY [timestamp -format %a ]
set DATE [timestamp -format %c ]

set send_human {.1 .3 1 .05 2}

#log_file $PATH/$SWITCH.expect.log
log_file /usr/local/etc/password.expect.log
log_user 0      # Disable logging to STDOUT
#log_user 1     # Enable logging to STDOUT

set timeout 10
spawn $TELNET $SWITCH
match_max 100000

expect "Trying"
expect {
   "Connected"  {

      expect "SW:v3.6" {
         send_log "\n\nThis version of software doesn't support the CLI commands!\n"
         send_user "\n\nThis version of software doesn't support the CLI commands!\n"
         exit 1
      }
      sleep 1
      send -- ""
                }
   Timeout      {
      send_log "We're unable to connect to the switch $SWITCH"
      send_user "We're unable to connect to the switch $SWITCH"
      exit 1;
                }
}

expect {
   "Username"   {
      send -- "RW\r"
   }
}

expect "Enter Password"
send -- "$PASSWORD\r"

expect {
   "Main Menu"  {
                }
   "Incorrect Password" {
      send_log "$SWITCH : Incorrect Password"
      exit 1
   }
   "Incorrect Credentials" {
      send_log "$SWITCH: Incorrect Credentials"
      exit 1
   }
}
sleep 1

# Let's get into the CLI interface from the menu prompts
send -- "C"

# Depending on the version of software we sometimes need a CR/LF
send -- "\r"
sleep 1

# Let's wait for the CLI prompt which includes the #
expect "#"
send -- "config term\r"
send -- "clock time-zone EST -5\r"
send -- "clock summer-time EDT date 9 Mar 2008 2:00 2 Nov 2008 2:00 +60\r"
send -- "exit\r"
send -- "logout\r"
expect eof

You can download the entire Expect script from this URL; set-nortel-timezone.exp.

The command line arguments are fairly straight forward;

usage: set-nortel-timezone.exp <SWITCH> <PASSWORD>

Where the SWITCH is the fully qualified domain name (FQDN) or the IP address of the switch in question and the PASSWORD is the Read-Write password for the switch.

If you had hundreds of switches to reconfigure you could wrap this Except script in a Bash shell script similar to the following;

#!/bin/bash
#
#####################################################################
#
# Language: Bash Shell Script
#
# Filename: /usr/local/etc/set-nortel-timezone.sh
#
# Purpose:  This script will kickoff the Expect script that will
#           configure the Daylight Saving Time features for each switch
#
# Author:   Michael McNamara
#
# Date:     June 1, 2008
#
# Version:  1.0
#
# Changes:
#
#           June 10, 2006 (M.McNamara)
#           -  added remote sites into shell script processing
#
#####################################################################
#

# Variables
PATH_TO=/usr/local/etc
UPGRADE=set-nortel-timezone.exp
MAIL_LIST=''
PAGER_LIST=''
ERROR_FLAG=0
MAILEXE='/usr/bin/mutt'
LOCKFILE=/tmp/trace.lck

# Check paramaters
if [ "$#" != 2 ]
then
  echo "Usage: `basename $0` <password>"
  exit 1
fi

PASSWORD=$1

#####################################################################
#####################################################################
# YOU SHOULD EDIT THE "SWITCHES" VARIABLE BELOW TO INCLUDE ALL THE
# SWITCHES THAT YOU WISH TO HAVE THE EXPECT SCRIPT RUN AGAINST
#####################################################################
#####################################################################

SWITCHES='sw1-5520.acme.org sw2-5520.acme.org sw3-5520.acme.org'

for SWITCH in $SWITCHES
do
        $PATH_TO/$UPGRADE $SWITCH $PASSWORD
done

exit

You can download the Bash shell script from this URL; set-nortel-timezone.sh.

I’ve only tested this on CentOS v5.2 but it should work on any Linux host with Expect installed although you may need to modify the path locations.

Cheers!

Network Time Protocol (NTP)

16

anclock

I’m sometimes amazed at how many large organizations don’t have a centralized Network Time Protocol (NTP) server setup and devices configured appropriately. When troubleshooting a problem it’s vital that the timestamps in the logs for each switch, router, server and appliance match up correctly.

I’m currently using two CentOS Linux servers to provide time services to over 10,000 devices in the network. My two servers are themselves syncing up with pool.ntp.org over the Internet. With CentOS I didn’t need to build the software, I only needed to install the NTP package through YUM and then configure it appropriately. It was really easy, much easier than it was say 10 years ago when you had to compile the NTP software (University of Delaware) by hand hoping you didn’t run into some missing library of version mismatch with the compiler.

We would first need to install the NTP software using YUM;
[root@hostname ]# yum install ntp

We would need to start the NTP daemons;
[root@hostname ]# service ntpd start

We would need to configure the server so the NTP software would start after every reboot;
[root@hostname ]# chkconfig ntpd on

With that step done we’d have ourselves and internal NTP server which would sync itself to the Internet (default configuration file in /etc/ntp.conf) and then our internal devices would sync to it.

Here are the CLI commands for configuring the ERS 8600 switch properly;

config bootconfig tz dst-name "EDT"
config bootconfig tz name "EST"
config bootconfig tz offset-from-utc 300
config bootconfig tz dst-end M11.1.0/0200
config bootconfig tz dst-start M3.2.0/0200

config ntp server create a.b.c.d
config ntp server create a.b.c.d
config ntp server create a.b.c.d
config ntp enable true

I’ve add the two configuration statements for the new Daylight Saving Time changes that were enacted in 2007. Please also note that I’m in the Eastern timezone (EDT/EST) so if you’re not in the Eastern timezone you would need to supplement your timezone abbreviation appropriately.

Here are the commands for an ES460,ES470,ERS4500 or ERS5500 series switch

5520-48T-PWR# config terminal
5520-48T-PWR (config)# sntp server primary a.b.c.d
5520-48T-PWR (config)# sntp server secondary a.b.c.d
5520-48T-PWR (config)# sntp enable
5520-48T-PWR (config)# exit5520-48T-PWR#

The ERS 4500/5500 Series now supports Daylight Saving Time. This feature is NOT supported on the ES460 and ES470 switches. --CORRECTION: this feature is support on the ES460/470 as of v3.7.x software, please see update at the bottom of this post for additional information. If you wanted to configure the timezone on the ERS4500/ERS5500 switch you would use the following commands;

5520-48T-PWR>enable
5520-48T-PWR# config terminal
5520-48T-PWR (config)# clock time-zone EST -5
5520-48T-PWR (config)# clock summer-time EDT date 9 Mar 2008 2:00 2 Nov 2008 2:00 +60
5520-48T-PWR (config)# exit
5520-48T-PWR#

You can use “show sntp” and “show clock” the ERS 5500 Series switch to check out your changes;

5530-24TFD#show sntp
SNTP Status:                      Enabled
Primary server address:         10.1.20.1
Secondary server address:     10.1.20.1
Sync interval:                      24 hours
Last sync source:                 10.1.20.1
Primary server sync failures:    0
Secondary server sync failures: 0
Last sync time:                  2008-06-14 14:47:31 GMT-04:00
Next sync time:                  2008-06-15 14:47:31 GMT-04:00
Current time:                     2008-06-15 13:52:24 GMT-04:00

5530-24TFD#show clock
Current SNTP time  :    2008-06-15 13:52:29 GMT-04:00
Summer time is set to:
start: 28 March 2007 at 02:00
end: 30 August 2008 at 15:00
Offset: 60 minutes. Timezone will be 'EDT'Time Zone is set to 'EST', offset from UTC is -05:00

Hopefully this will provide a brief look into NTP,SNTP and you’ll agree that it really isn’t that hard to setup and configure properly.

Cheers!

Update: June 17, 2008

After posting the article above I decided I would confirm that the Daylight Saving Time feature was not available on the Nortel Ethernet Switch 460/470. I found that as of v3.7.x software the feature is supported on the switches. The configuration commands are identical to the ERS4500/ERS5500 switches. Here’s an example specifically for the Eastern timezone.

470-48T>enable470-48T#config term
Enter configuration commands, one per line.  End with CNTL/Z.
470-48T(config)#clock time-zone EST -5 00
470-48T(config)#clock summer-time EDT date 9 Mar 2008 02:00 2 Nov 2008 2:00 +60
470-48T(config)#show clock summer-time
Summer time is set to:start: 9 March 2008 at 02:00end: 2 November 2008 at 02:00
Offset: 60 minutes. Timezone will be 'EDT'
470-48T(config)#exit

Cheers!

How much uptime is too much?

7

A quick story for everyone…

We generally perform software upgrades on all our routers and switches twice a year. It really helps to keep our network infrastructure current and it also helps to reduced unscheduled downtime.

Last fall we decided to skip the bi-yearly maintenance because there were just too many projects on the docket. This spring we came across a very interesting issue that we had never seen in the past. We started to notice that multiple Nortel Ethernet Switch 460/470 switches/stacks were rebooting themselves all over our network. It took us a few hours to realize that every switch that had rebooted had just eclipsed approximately 500 days of uptime. All the affected switches were running FW 3.6.0.6 with SW v3.6.4.08. The switches were literally rebooting themselves in the same order in which they had been upgraded almost 500 days earlier.

I’m currently trying to confirm with Nortel that this “bug” has been removed from the 3.7.x software release.

This was one occasion where the network was just too good for itself.

Cheers!

Update: Tuesday June 10, 2008

I received a formal response from Nortel today that included the following:

Analysis of the issue :-
When the BS-470 switches reaches 497 days the system time rolls over and during this period management communication will be lost. This is caused by the use of a 32 bit counter, which when it rolls back to 0, initiates an internal software synchronization to align all timers. This is only loss of IP management and not switching functionality.

This issue still open and can be fixed by rebooting the switches before reaching the 497 day mark.

When I inquired if the problem had been resolved in the v3.7.x software release I was told it had not. It would seem that a lot of folks just don’t expect switches to be running that long these days.

Cheers!

Update: Wednesday November 4, 2008

Last week Nortel released a technical service bulletin entitled, “Ethernet Routing Switches: SysUpTime approaching 497 days can cause the switch or stack to behave in some unexpected way“. They also released a video that documents a workaround to the problem.

Let me save you the time and effort of downloading either. Nortel solution is truely masterful; reboot the switch.

While I’ve been know to defend Nortel there’s just no defense for this. I’m completely floored at Nortel’s response.

Cheers!

Go to Top